[jira] [Commented] (IMAGING-369) TIFF JPEG reader encounters array bounds exception on edge cases.
[ https://issues.apache.org/jira/browse/IMAGING-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788435#comment-17788435 ] Gary Lucas commented on IMAGING-369: Changes now integrated into master branch at Github. > TIFF JPEG reader encounters array bounds exception on edge cases. > - > > Key: IMAGING-369 > URL: https://issues.apache.org/jira/browse/IMAGING-369 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > Fix For: 1.0 > > > The changes recently introduced for Imaging-194 permit the processing of TIFF > files that use JPEG compression internally (these are TIFF files, not JPEG > files). However, there is an edge case where they can fail and throw an > array-bounds exception. This Issue proposes to fix the bug and contribute an > additional test file that exercises the section of code that has the problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-369) TIFF JPEG reader encounters array bounds exception on edge cases.
[ https://issues.apache.org/jira/browse/IMAGING-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787240#comment-17787240 ] Gary Lucas commented on IMAGING-369: Pull request #335 submitted with fix for this problem. > TIFF JPEG reader encounters array bounds exception on edge cases. > - > > Key: IMAGING-369 > URL: https://issues.apache.org/jira/browse/IMAGING-369 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > The changes recently introduced for Imaging-194 permit the processing of TIFF > files that use JPEG compression internally (these are TIFF files, not JPEG > files). However, there is an edge case where they can fail and throw an > array-bounds exception. This Issue proposes to fix the bug and contribute an > additional test file that exercises the section of code that has the problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IMAGING-369) TIFF JPEG reader encounters array bounds exception on edge cases.
Gary Lucas created IMAGING-369: -- Summary: TIFF JPEG reader encounters array bounds exception on edge cases. Key: IMAGING-369 URL: https://issues.apache.org/jira/browse/IMAGING-369 Project: Commons Imaging Issue Type: Bug Components: Format: TIFF Affects Versions: 1.x Reporter: Gary Lucas The changes recently introduced for Imaging-194 permit the processing of TIFF files that use JPEG compression internally (these are TIFF files, not JPEG files). However, there is an edge case where they can fail and throw an array-bounds exception. This Issue proposes to fix the bug and contribute an additional test file that exercises the section of code that has the problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-97) Fails to read most Sony A100 JPG files generated from Photoshop CS2
[ https://issues.apache.org/jira/browse/IMAGING-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787232#comment-17787232 ] Gary Lucas commented on IMAGING-97: --- I just entered Issue-368 describing problems I've seen with the file _DSC6099.jpg that was contributed to Imaging's JUnit test suite following on this Jira item. The image reader doesn't crash, but it fails to properly decode the image. If anyone has suggestions, I might have time to look at this. > Fails to read most Sony A100 JPG files generated from Photoshop CS2 > > > Key: IMAGING-97 > URL: https://issues.apache.org/jira/browse/IMAGING-97 > Project: Commons Imaging > Issue Type: New Feature > Components: Format: JPEG >Affects Versions: 1.0-alpha1 > Environment: Windows 7, JDK1.7, Imaging snapshot build from 20121013 >Reporter: William Saar >Priority: Major > Fix For: Patch Needed > > Attachments: _DSC6099.jpg > > > This line fails with the exception at the bottom for most of my old JPG files > shot with a Sony A100, and sometimes processed by Photoshop CS2. I will > attach a failing file. > BufferedImage img = Imaging.getBufferedImage(compressedThumbnailData, > jpgDecodeParams); > where jpgDecodeParams are > ImagingConstants.BUFFERED_IMAGE_FACTORY, new RgbBufferedImageFactory() > ImagingConstants.PARAM_KEY_FORMAT, ImageFormat.IMAGE_FORMAT_JPEG > Exception: > org.apache.commons.imaging.ImageReadException: Invalid marker found in > entropy data > at > org.apache.commons.imaging.formats.jpeg.decoder.JpegInputStream.nextBit(JpegInputStream.java:50) > at > org.apache.commons.imaging.formats.jpeg.decoder.JpegDecoder.decode(JpegDecoder.java:417) > at > org.apache.commons.imaging.formats.jpeg.decoder.JpegDecoder.readMCU(JpegDecoder.java:311) > at > org.apache.commons.imaging.formats.jpeg.decoder.JpegDecoder.visitSOS(JpegDecoder.java:122) > at > org.apache.commons.imaging.formats.jpeg.JpegUtils.traverseJFIF(JpegUtils.java:83) > at > org.apache.commons.imaging.formats.jpeg.decoder.JpegDecoder.decode(JpegDecoder.java:428) > at > org.apache.commons.imaging.formats.jpeg.JpegImageParser.getBufferedImage(JpegImageParser.java:95) > at > org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1375) > at > org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1315) > at -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IMAGING-368) JPEG test results in fragmented image
Gary Lucas created IMAGING-368: -- Summary: JPEG test results in fragmented image Key: IMAGING-368 URL: https://issues.apache.org/jira/browse/IMAGING-368 Project: Commons Imaging Issue Type: Bug Affects Versions: 1.x Environment: !Comparison_DSC6099.jpg! Reporter: Gary Lucas Attachments: Comparison_DSC6099.jpg When reading JPEG file _DSC6099.jpg from Commons Imaging's test image data set, Imaging results in a broken image. This problem may be related to issues reported in Imaging-97. I've attached an example comparing the results when using ImageIO versus those when using Commons Imaging. I tried this same test with earlier versions of Commons Imaging and got the same results. So it does not appear to be due to recent changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-194) Tiff with JPEG,Zip compression fails to decompress
[
https://issues.apache.org/jira/browse/IMAGING-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786054#comment-17786054
]
Gary Lucas commented on IMAGING-194:
New code to address this issue is now available in Commons Imaging
> Tiff with JPEG,Zip compression fails to decompress
> --
>
> Key: IMAGING-194
> URL: https://issues.apache.org/jira/browse/IMAGING-194
> Project: Commons Imaging
> Issue Type: Improvement
> Components: Format: TIFF
>Affects Versions: 1.0-alpha1
>Reporter: Satya Deep Maheshwari
>Priority: Major
>
> Tiff with JPEG, Zip compression fails to decompress with the below exception:
> {code}
> org.apache.commons.imaging.ImageReadException: Tiff: unknown/unsupported
> compression: 7
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReader.decompress(DataReader.java:215)
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReaderStrips.readImageData(DataReaderStrips.java:210)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:650)
> at
> org.apache.commons.imaging.formats.tiff.TiffDirectory.getTiffImage(TiffDirectory.java:157)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:463)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1407)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1370)
> {code}
> From the
> [documentation|https://commons.apache.org/proper/commons-imaging/formatsupport.html]
> , it seems this compression format is not supported. Excerpt from the
> document below:
> {quote}
> Supported through version 6.0. TIFFs is a open-ended container format, so
> it's not possible to support every possibly variation. Supports Bi-Level,
> Palette/Indexed, RGB, CMYK, YCbCr, CIELab and LOGLUV images. Supports reading
> and writing LZW, CCITT Modified Huffman/Group 3/Group 4, and Packbits/RLE
> compression. Notably missing other forms of compression though, including
> JPEG. Supports reading Tiled images.
> {quote}
> This ticket is logged to add JPEG/Zip compression format support.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-194) Tiff with JPEG,Zip compression fails to decompress
[
https://issues.apache.org/jira/browse/IMAGING-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785258#comment-17785258
]
Gary Lucas commented on IMAGING-194:
I created test files using LibTIFF and ImageMagick.
Just submitted pull request 334. This change enables the reading of TIFF files
that use JPEG format for their internal coding.
Changes to be able to write files in this format will have to wait. I am
thinking that would be a separate issue.
> Tiff with JPEG,Zip compression fails to decompress
> --
>
> Key: IMAGING-194
> URL: https://issues.apache.org/jira/browse/IMAGING-194
> Project: Commons Imaging
> Issue Type: Improvement
> Components: Format: TIFF
>Affects Versions: 1.0-alpha1
>Reporter: Satya Deep Maheshwari
>Priority: Major
>
> Tiff with JPEG, Zip compression fails to decompress with the below exception:
> {code}
> org.apache.commons.imaging.ImageReadException: Tiff: unknown/unsupported
> compression: 7
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReader.decompress(DataReader.java:215)
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReaderStrips.readImageData(DataReaderStrips.java:210)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:650)
> at
> org.apache.commons.imaging.formats.tiff.TiffDirectory.getTiffImage(TiffDirectory.java:157)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:463)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1407)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1370)
> {code}
> From the
> [documentation|https://commons.apache.org/proper/commons-imaging/formatsupport.html]
> , it seems this compression format is not supported. Excerpt from the
> document below:
> {quote}
> Supported through version 6.0. TIFFs is a open-ended container format, so
> it's not possible to support every possibly variation. Supports Bi-Level,
> Palette/Indexed, RGB, CMYK, YCbCr, CIELab and LOGLUV images. Supports reading
> and writing LZW, CCITT Modified Huffman/Group 3/Group 4, and Packbits/RLE
> compression. Notably missing other forms of compression though, including
> JPEG. Supports reading Tiled images.
> {quote}
> This ticket is logged to add JPEG/Zip compression format support.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-362) TIFF tags 0x111 and 0x117 have wrong descriptive names
[ https://issues.apache.org/jira/browse/IMAGING-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782892#comment-17782892 ] Gary Lucas commented on IMAGING-362: I've been away for awhile, so I don't know when this happened... But it appears that Imaging is using the ExifTagConstants class rather than TiffTagConstants when reading these elements. Perhaps it was always that way. The ExifTagConstants has the same mistake as the code I fixed. I will investigate further. I don't think that having redundant specifications is ideal, but I don't know what the level-of-effort/potential-to-break-things would be for trying to unify the code. At the very least, I will update ExifTagConstants to be correct. > TIFF tags 0x111 and 0x117 have wrong descriptive names > -- > > Key: IMAGING-362 > URL: https://issues.apache.org/jira/browse/IMAGING-362 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0-alpha2 >Reporter: Gary Lucas >Priority: Minor > Fix For: 1.0 > > > The TIFF standard defines metadata elements using integer code values. For > code values 0x111 and 0x117, Commons Imaging associates these with incorrect > names "PreviewImageStart" and "PreviewImageLength". According to the > specification "TIFF Revision 6.0 Final – June 3, 1992", the correct names for > these are "StripOffsets" and "StripByteCounts". In fact, the word "preview" > never even shows up in the TIFF specification. > This should be an easy fix. No new test cases will be required, since these > are descriptive elements only. > The role of these two elements is to define the file position and length of > data elements for TIFF files that are encoded as strips. > One of the example programs in the Commons Imaging distribution (which is not > compiled) is called ReadTagsAndImages.java. It can be used to dump TIFF tags. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-194) Tiff with JPEG,Zip compression fails to decompress
[
https://issues.apache.org/jira/browse/IMAGING-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770433#comment-17770433
]
Gary Lucas commented on IMAGING-194:
I've got an implementation of this feature ready to go (TIFF images with
internal JPEG encoding). But I need test files. The Commons Imaging team
requires that changes come with valid JUnit tests. To write JUnit tests (which
I am happy to do), I need some sample files that can be included in the code
distribution. So they need to be pretty small and free of copyright
constraints (less than 100 KB is good, less than 25 KB is ideal).
Unfortunately, all of the sample files I have are quite large.
Does anyone have a source of sample files they'd be willing to contribute?
> Tiff with JPEG,Zip compression fails to decompress
> --
>
> Key: IMAGING-194
> URL: https://issues.apache.org/jira/browse/IMAGING-194
> Project: Commons Imaging
> Issue Type: Improvement
> Components: Format: TIFF
>Affects Versions: 1.0-alpha1
>Reporter: Satya Deep Maheshwari
>Priority: Major
>
> Tiff with JPEG, Zip compression fails to decompress with the below exception:
> {code}
> org.apache.commons.imaging.ImageReadException: Tiff: unknown/unsupported
> compression: 7
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReader.decompress(DataReader.java:215)
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReaderStrips.readImageData(DataReaderStrips.java:210)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:650)
> at
> org.apache.commons.imaging.formats.tiff.TiffDirectory.getTiffImage(TiffDirectory.java:157)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:463)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1407)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1370)
> {code}
> From the
> [documentation|https://commons.apache.org/proper/commons-imaging/formatsupport.html]
> , it seems this compression format is not supported. Excerpt from the
> document below:
> {quote}
> Supported through version 6.0. TIFFs is a open-ended container format, so
> it's not possible to support every possibly variation. Supports Bi-Level,
> Palette/Indexed, RGB, CMYK, YCbCr, CIELab and LOGLUV images. Supports reading
> and writing LZW, CCITT Modified Huffman/Group 3/Group 4, and Packbits/RLE
> compression. Notably missing other forms of compression though, including
> JPEG. Supports reading Tiled images.
> {quote}
> This ticket is logged to add JPEG/Zip compression format support.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-362) TIFF tags 0x111 and 0x117 have wrong descriptive names
[ https://issues.apache.org/jira/browse/IMAGING-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769687#comment-17769687 ] Gary Lucas commented on IMAGING-362: Hi Gary, Thanks for taking care of this (and the others you just addressed). Gary (the other Gary) > TIFF tags 0x111 and 0x117 have wrong descriptive names > -- > > Key: IMAGING-362 > URL: https://issues.apache.org/jira/browse/IMAGING-362 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0-alpha2 >Reporter: Gary Lucas >Priority: Minor > Fix For: 1.0 > > > The TIFF standard defines metadata elements using integer code values. For > code values 0x111 and 0x117, Commons Imaging associates these with incorrect > names "PreviewImageStart" and "PreviewImageLength". According to the > specification "TIFF Revision 6.0 Final – June 3, 1992", the correct names for > these are "StripOffsets" and "StripByteCounts". In fact, the word "preview" > never even shows up in the TIFF specification. > This should be an easy fix. No new test cases will be required, since these > are descriptive elements only. > The role of these two elements is to define the file position and length of > data elements for TIFF files that are encoded as strips. > One of the example programs in the Commons Imaging distribution (which is not > compiled) is called ReadTagsAndImages.java. It can be used to dump TIFF tags. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IMAGING-364) NullPointerException in TiffDirectory.getTiffImage()
[
https://issues.apache.org/jira/browse/IMAGING-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Lucas updated IMAGING-364:
---
Description:
When calling TiffDirectory.getTiffImage(), Commons Imaging throws a null
pointer exception.
{code:java}
java.lang.NullPointerException
at
org.apache.commons.imaging.formats.tiff.TiffImageParser.checkForSubImage(TiffImageParser.java:78)
at
org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:261)
at
org.apache.commons.imaging.formats.tiff.TiffDirectory.getTiffImage(TiffDirectory.java:776){code}
If you call the alternate method
TiffDirectory.getTiffImage(TiffImagingParameters), it works okay. The problem
is that the internal method TiffImageParser().getBufferedImage() cannot handle
a null reference.
Here's the problematic code from TiffDirectory.java
{code:java}
public BufferedImage getTiffImage() throws ImagingException, IOException {
if (null == abstractTiffImageData) {
return null;
}
return new TiffImageParser().getBufferedImage(
this, headerByteOrder, null);
}
{code}
We could either make TiffImageParser more robust, or we could avoid passing in
a null value for the third argument.
{code:java}
TiffImagingParameters defaultParameters = new TiffImagingParameters();
return new TiffImageParser().getBufferedImage(
this, headerByteOrder, defaultParameters);{code}
was:
When calling TiffDirectory.getTiffImage(), Commons Imaging throws a null
pointer exception. If you call the alternate method
TiffDirectory.getTiffImage(TiffImagingParameters), it works okay. The problem
is that the internal method TiffImageParser().getBufferedImage() cannot handle
a null reference.
Here's the problematic code.
{code:java}
public BufferedImage getTiffImage() throws ImagingException, IOException {
if (null == abstractTiffImageData) {
return null;
}
return new TiffImageParser().getBufferedImage(
this, headerByteOrder, null);
}
{code}
We could either make TiffImageParser more robust, or we could avoid passing in
a null value for the third argument.
{code:java}
TiffImagingParameters defaultParameters = new TiffImagingParameters();
return new TiffImageParser().getBufferedImage(
this, headerByteOrder, defaultParameters);{code}
> NullPointerException in TiffDirectory.getTiffImage()
>
>
> Key: IMAGING-364
> URL: https://issues.apache.org/jira/browse/IMAGING-364
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Affects Versions: 1.0-alpha3
>Reporter: Gary Lucas
>Priority: Major
>
> When calling TiffDirectory.getTiffImage(), Commons Imaging throws a null
> pointer exception.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.checkForSubImage(TiffImageParser.java:78)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:261)
> at
> org.apache.commons.imaging.formats.tiff.TiffDirectory.getTiffImage(TiffDirectory.java:776){code}
>
> If you call the alternate method
> TiffDirectory.getTiffImage(TiffImagingParameters), it works okay. The
> problem is that the internal method TiffImageParser().getBufferedImage()
> cannot handle a null reference.
>
> Here's the problematic code from TiffDirectory.java
> {code:java}
> public BufferedImage getTiffImage() throws ImagingException, IOException {
> if (null == abstractTiffImageData) {
> return null;
> }
> return new TiffImageParser().getBufferedImage(
> this, headerByteOrder, null);
> }
> {code}
>
> We could either make TiffImageParser more robust, or we could avoid passing
> in a null value for the third argument.
>
> {code:java}
> TiffImagingParameters defaultParameters = new TiffImagingParameters();
> return new TiffImageParser().getBufferedImage(
> this, headerByteOrder, defaultParameters);{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IMAGING-364) NullPointerException in TiffDirectory.getTiffImage()
Gary Lucas created IMAGING-364:
--
Summary: NullPointerException in TiffDirectory.getTiffImage()
Key: IMAGING-364
URL: https://issues.apache.org/jira/browse/IMAGING-364
Project: Commons Imaging
Issue Type: Bug
Components: Format: TIFF
Affects Versions: 1.0-alpha3
Reporter: Gary Lucas
When calling TiffDirectory.getTiffImage(), Commons Imaging throws a null
pointer exception. If you call the alternate method
TiffDirectory.getTiffImage(TiffImagingParameters), it works okay. The problem
is that the internal method TiffImageParser().getBufferedImage() cannot handle
a null reference.
Here's the problematic code.
{code:java}
public BufferedImage getTiffImage() throws ImagingException, IOException {
if (null == abstractTiffImageData) {
return null;
}
return new TiffImageParser().getBufferedImage(
this, headerByteOrder, null);
}
{code}
We could either make TiffImageParser more robust, or we could avoid passing in
a null value for the third argument.
{code:java}
TiffImagingParameters defaultParameters = new TiffImagingParameters();
return new TiffImageParser().getBufferedImage(
this, headerByteOrder, defaultParameters);{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-362) TIFF tags 0x111 and 0x117 have wrong descriptive names
[
https://issues.apache.org/jira/browse/IMAGING-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764633#comment-17764633
]
Gary Lucas commented on IMAGING-362:
Here's an example from ReadTagsAndImages.java
{code:java}
273 (0x111: PreviewImageStart): 2228, 2392, 2552, 2716, 2884, 3044, 3200,
3364, 3520, 3660
277 (0x115: SamplesPerPixel): 3 (1 Short)
278 (0x116: RowsPerStrip): 3 (1 Long)
279 (0x117: PreviewImageLength): 164, 160, 162, 165, 157, 155, 162, 156, 138,
172, 202, 22
{code}
> TIFF tags 0x111 and 0x117 have wrong descriptive names
> --
>
> Key: IMAGING-362
> URL: https://issues.apache.org/jira/browse/IMAGING-362
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Affects Versions: 1.0-alpha2
>Reporter: Gary Lucas
>Priority: Minor
> Fix For: 1.0-alpha3
>
>
> The TIFF standard defines metadata elements using integer code values. For
> code values 0x111 and 0x117, Commons Imaging associates these with incorrect
> names "PreviewImageStart" and "PreviewImageLength". According to the
> specification "TIFF Revision 6.0 Final – June 3, 1992", the correct names for
> these are "StripOffsets" and "StripByteCounts". In fact, the word "preview"
> never even shows up in the TIFF specification.
> This should be an easy fix. No new test cases will be required, since these
> are descriptive elements only.
> The role of these two elements is to define the file position and length of
> data elements for TIFF files that are encoded as strips.
> One of the example programs in the Commons Imaging distribution (which is not
> compiled) is called ReadTagsAndImages.java. It can be used to dump TIFF tags.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IMAGING-362) TIFF tags 0x111 and 0x117 have wrong descriptive names
Gary Lucas created IMAGING-362: -- Summary: TIFF tags 0x111 and 0x117 have wrong descriptive names Key: IMAGING-362 URL: https://issues.apache.org/jira/browse/IMAGING-362 Project: Commons Imaging Issue Type: Bug Components: Format: TIFF Affects Versions: 1.0-alpha2 Reporter: Gary Lucas Fix For: 1.0-alpha3 The TIFF standard defines metadata elements using integer code values. For code values 0x111 and 0x117, Commons Imaging associates these with incorrect names "PreviewImageStart" and "PreviewImageLength". According to the specification "TIFF Revision 6.0 Final – June 3, 1992", the correct names for these are "StripOffsets" and "StripByteCounts". In fact, the word "preview" never even shows up in the TIFF specification. This should be an easy fix. No new test cases will be required, since these are descriptive elements only. The role of these two elements is to define the file position and length of data elements for TIFF files that are encoded as strips. One of the example programs in the Commons Imaging distribution (which is not compiled) is called ReadTagsAndImages.java. It can be used to dump TIFF tags. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764433#comment-17764433 ] Gary Lucas commented on IMAGING-316: Submitted Pull Request 318 to the Commons Imaging project on github. This enhancement will allow the Imaging library to read BigTIFF format files. The ability to write BigTIFF files will be subject of a future Jira issue. > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types. And it is > possible that a file address with the high bit set might be incorrectly > interpreted as a negative number. So I will be taking a look at the code to > make sure all file addresses are properly masked when they are handed over to > Java. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763473#comment-17763473 ] Gary Lucas commented on IMAGING-316: Have an updated version of the Commons Imaging library that successfully reads all of the sample images at the AWare Systems website. Am working on the JUnits tests. I expect to submit a pull request in the next few days. > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types. And it is > possible that a file address with the high bit set might be incorrectly > interpreted as a negative number. So I will be taking a look at the code to > make sure all file addresses are properly masked when they are handed over to > Java. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762356#comment-17762356 ] Gary Lucas commented on IMAGING-316: I requested and received permission from Joris Van Damme (of AWare Systems) to include the sample BigTIFF files I mentioned in the Commons Imaging distribution under the test resources files. These samples are a set of 8 BigTIFF files and one conventional TIFF file provided for reference purposes. The files are relatively small, making the whole collection come in at 158 KB (the "big" refers to the 64-bit address space, not the actual size of the files). This collection exercises all of the core functions that are affected by the 64-bit address space and will provide a solid basis for JUnit tests. I will, of course, include a proper citation and grateful acknowledgement of the data source in a README.TXT file. To learn more about BigTIFF, visit Mr. Van Damme's page at [https://www.awaresystems.be/imaging/tiff/bigtiff.html] > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types. And it is > possible that a file address with the high bit set might be incorrectly > interpreted as a negative number. So I will be taking a look at the code to > make sure all file addresses are properly masked when they are handed over to > Java. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762228#comment-17762228 ] Gary Lucas commented on IMAGING-316: I've located a great page with resources for the BigTIFF file format. AWare Systems has posted a page on the [The BigTIFF File format|https://www.awaresystems.be/imaging/tiff/bigtiff.html] that describes the enhancements required to support BigTIFF and provides a comprehensive set of test tiles to exercise any API that reads BigTIFF. These resources will be a great boost in developing this feature for Commons Imaging. > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types. And it is > possible that a file address with the high bit set might be incorrectly > interpreted as a negative number. So I will be taking a look at the code to > make sure all file addresses are properly masked when they are handed over to > Java. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-361) ImageInfo reports wrong color-type for non-RGB TIFF files
[
https://issues.apache.org/jira/browse/IMAGING-361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758600#comment-17758600
]
Gary Lucas commented on IMAGING-361:
Just found that this issue repeats previously reported issue Imaging-337.
> ImageInfo reports wrong color-type for non-RGB TIFF files
> --
>
> Key: IMAGING-361
> URL: https://issues.apache.org/jira/browse/IMAGING-361
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Reporter: Gary Lucas
>Priority: Minor
>
> When TiffImageParser.getImageInfo() creates an ImageInfo object, it has a
> hard-wired value for ColorType of RGB. So images with color models such as
> CMYK or YCbCr are reported as RGB. It should look at the Photometric
> Interpretation tag and assign a value accordingly.
> Note that this issue has nothing to do with the kind of BufferedImage that
> other methods will produce. A code fix would only affect the values
> reported in ImageInfo.
> The problem code is currently at line 534 in TiffImageParser.java, but
> examples of accessing other TIFF tags are located nearby in the code.
> {code:java}
> final ImageInfo.ColorType colorType = ImageInfo.ColorType.RGB
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IMAGING-361) ImageInfo reports wrong color-type for non-RGB TIFF files
[
https://issues.apache.org/jira/browse/IMAGING-361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Lucas updated IMAGING-361:
---
Description:
When TiffImageParser.getImageInfo() creates an ImageInfo object, it has a
hard-wired value for ColorType of RGB. So images with color models such as
CMYK or YCbCr are reported as RGB. It should look at the Photometric
Interpretation tag and assign a value accordingly.
Note that this issue has nothing to do with the kind of BufferedImage that
other methods will produce. A code fix would only affect the values reported
in ImageInfo.
The problem code is currently at line 534 in TiffImageParser.java, but examples
of accessing other TIFF tags are located nearby in the code.
{code:java}
final ImageInfo.ColorType colorType = ImageInfo.ColorType.RGB
{code}
was:
When TiffImageParser.getImageInfo() creates an ImageInfo object, it has a
hard-wired value for ColorType of RGB. So images with color models such as
CMYK or YCbCr are reported as RGB. It should look at the Photometric
Interpretation tag and assign a value accordingly.
Note that this issue has nothing to do with the kind of BufferedImage that
other methods will produce. A code fix would only affect the values reported
in ImageInfo.
The problem code is currently at line 534 in TiffImageParser.java, but examples
of accessing other TIFF tags are located nearby in the code.
{code:java}
final ImageInfo.ColorType colorType = ImageInfo.ColorType.RGB
{code}
;
> ImageInfo reports wrong color-type for non-RGB TIFF files
> --
>
> Key: IMAGING-361
> URL: https://issues.apache.org/jira/browse/IMAGING-361
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Reporter: Gary Lucas
>Priority: Minor
>
> When TiffImageParser.getImageInfo() creates an ImageInfo object, it has a
> hard-wired value for ColorType of RGB. So images with color models such as
> CMYK or YCbCr are reported as RGB. It should look at the Photometric
> Interpretation tag and assign a value accordingly.
> Note that this issue has nothing to do with the kind of BufferedImage that
> other methods will produce. A code fix would only affect the values
> reported in ImageInfo.
> The problem code is currently at line 534 in TiffImageParser.java, but
> examples of accessing other TIFF tags are located nearby in the code.
> {code:java}
> final ImageInfo.ColorType colorType = ImageInfo.ColorType.RGB
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IMAGING-361) ImageInfo reports wrong color-type for non-RGB TIFF files
Gary Lucas created IMAGING-361:
--
Summary: ImageInfo reports wrong color-type for non-RGB TIFF files
Key: IMAGING-361
URL: https://issues.apache.org/jira/browse/IMAGING-361
Project: Commons Imaging
Issue Type: Bug
Components: Format: TIFF
Reporter: Gary Lucas
When TiffImageParser.getImageInfo() creates an ImageInfo object, it has a
hard-wired value for ColorType of RGB. So images with color models such as
CMYK or YCbCr are reported as RGB. It should look at the Photometric
Interpretation tag and assign a value accordingly.
Note that this issue has nothing to do with the kind of BufferedImage that
other methods will produce. A code fix would only affect the values reported
in ImageInfo.
The problem code is currently at line 534 in TiffImageParser.java, but examples
of accessing other TIFF tags are located nearby in the code.
{code:java}
final ImageInfo.ColorType colorType = ImageInfo.ColorType.RGB
{code}
;
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-360) Missing some TIFF compression information in ImageInfo
[ https://issues.apache.org/jira/browse/IMAGING-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755302#comment-17755302 ] Gary Lucas commented on IMAGING-360: Submitted Pull Request #311 to github > Missing some TIFF compression information in ImageInfo > -- > > Key: IMAGING-360 > URL: https://issues.apache.org/jira/browse/IMAGING-360 > Project: Commons Imaging > Issue Type: Bug >Reporter: Gary Lucas >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-360) Missing some TIFF compression information in ImageInfo
[
https://issues.apache.org/jira/browse/IMAGING-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755300#comment-17755300
]
Gary Lucas commented on IMAGING-360:
The ImageInfo for TIFF is missing information related to three TIFF compression
options, two of which are quite commonly used. I propose to modify the
TiffImageParser class to populate these values.
{code:java}
TIFF_COMPRESSION_DEFLATE_PKZIP
TIFF_COMPRESSION_DEFLATE_ADOBE {code}
Additionally, the TIFF format has an obsolete variation on JPEG that was
abandoned in the 1990's. Commons Imaging does not currently support TIFF JPEG
compression, though many of the pieces needed are already in its code base. I
propose to do the following:
The current code has a class called TiffConstants that includes the definition
for
{code:java}
TIFF_COMPRESSION_JPEG = 6;{code}
Code 6 is the obsolete JPEG variation. Code 7 is the format currently used in
many TIFF utilities. So I propose to modify the constants and as follows
{code:java}
TIFF_COMPRESSION_JPEG_OBSOLETE = 6
TIFF_COMPRESSION_JPEG = 7
{code}
I will update ImageInfo's inner enum "CompressionAlgorithm" to have a support
entry for JPEG_TIFF_OBSOLETE (it already has one for JPEG). So anyone who gets
ImageInfo on a legacy TIFF file will at least get meaningful information about
its format.
> Missing some TIFF compression information in ImageInfo
> --
>
> Key: IMAGING-360
> URL: https://issues.apache.org/jira/browse/IMAGING-360
> Project: Commons Imaging
> Issue Type: Bug
>Reporter: Gary Lucas
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IMAGING-360) Missing some TIFF compression information in ImageInfo
Gary Lucas created IMAGING-360: -- Summary: Missing some TIFF compression information in ImageInfo Key: IMAGING-360 URL: https://issues.apache.org/jira/browse/IMAGING-360 Project: Commons Imaging Issue Type: Bug Reporter: Gary Lucas -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748957#comment-17748957 ] Gary Lucas commented on IMAGING-356: That's certainly feasible within the structure of the existing code. In any case, an exception would be an exception, but at least we could make the code throw an exception that provided slightly more information to the developer using the API (something like "your image file has to be broken because otherwise this stuff would work"). > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > Attachments: image-2023-07-04-08-52-36-535.png > > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748846#comment-17748846 ] Gary Lucas commented on IMAGING-356: In follow up to my 17 July post, I made several attempts to expedite processing by making code changes. None of them represented a significant improvement and several "obvious improvements" actually degraded performance. My thought is that any effective optimizations are already covered by Java's impressive JIT compiler and optimizer. TIFF files store their data in blocks (strips or tiles), but even changing the code to operate on block-oriented structures did not help. My guess is that the more complicated code just got in the way of Java's in-lining and other optimizations. Also, I think alternate data structures might have interfered with the allocation of L2/L3 fast cache memory, though this is an area in which I do not have a lot of experience. I'm willing to abide by the consensus of the community here But If I had a vote on this one, I'd like to get rid of the Math.addExtact and Math.multipleExact methods since I think that these are just overcautious coding practices. For one thing, the conditions they handle will never happen (if something. corrupted the source image file, the Imaging code will fail long before it tries to store pixel values in a memory). And for another, even if the runtime processes made it into that function, the strict math doesn't correct a problem. It just throws a different exception that the native code would do. In fact, I think a case could be made for getting rid of the check on the index value and going with the bare-bones implementation. In that case, the worst thing that would happen is that the code would throw an ArrayIndexOutOfBounds exception rather than the IllegalArgumentException or ArithmeticException. And just FYI... The fastest test results I got were with the bare bones expression shown below. I think that by avoiding computing the index in a separate variable, Java can optimize the statement slightly better than otherwise. data[ y * width + x] = argb; > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > Attachments: image-2023-07-04-08-52-36-535.png > > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743841#comment-17743841
]
Gary Lucas commented on IMAGING-356:
Gary,
I think you are correct in identifying ImageBuilder.setRGB() as a bottleneck.
The setRGB method is called to populate each pixel in an image one-at-a-time.
I will look at what would be required to implement a method that populated a
set of multiple pixels (perhaps one row at a time).
The current version uses the Java Math.multiplyExact and Math.addExact methods
to perform exact arithmetic in an effort to avoid integer overflow. Of course,
integer overflow is a condition that will "never" happen... The elements
involved, image width and x/y coordinates should be well within the range of
normal integer arithmetic. And if they're not, then something else will break
long before setRGB is called. Even so, I am not strongly advocating changing
setRGB back to the form I originally implemented.
Just to gain insights, I looked at reverting the code back to its original form.
The current code looks something like:
{code:java}
final int rowOffset = Math.multiplyExact(y, width);
final int index = Math.addExact(rowOffset, x);
if (index > data.length) {
throw new IllegalArgumentException("setRGB: Illegal array index.");
}
data[index] = argb;{code}
I tried refactoring it two ways (bare bones):
{code:java}
data[y * width + x] = argb;{code}
And range checked
{code:java}
final int index = y * width + x;
if(0<=index && index TIFF reading extremely slow in version 1.0-SNAPSHOT
> ---
>
> Key: IMAGING-356
> URL: https://issues.apache.org/jira/browse/IMAGING-356
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Affects Versions: 1.0
>Reporter: Gary Lucas
>Priority: Major
> Attachments: image-2023-07-04-08-52-36-535.png
>
>
> I am using the latest code from github (1.0-SNAPSHOT downloaded from github
> of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required
> 673 milliseconds to read that file. The new code requires upward of 15
> minutes. Clearly something got broken since the last release.
> The TIFF file is a 1x1 pixel 4 byte image format organized in strips.
> The bottleneck appears to occur in the TiffReader getTiffRawImageData method
> which reads raw data from the file in preparation of creating a BufferedImage
> object.
> I suspect that there may be a general slowness of file access. In debugging,
> even reading the initial metadata (22 TIFF tags) took a couple of seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (IMAGING-358) "RationalNumber" class is "public"
[ https://issues.apache.org/jira/browse/IMAGING-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743803#comment-17743803 ] Gary Lucas edited comment on IMAGING-358 at 7/17/23 1:17 PM: - So what you are proposing is to replace code that has been around for long time with an alternate module. I cited concerns that the alternate approach might not work due to the possibility of difference in implementation and rules. You asked the reasonable question "where are the differences?" And now I will make the reasonable observation that the onus is on you to answer that, not me. After all, you are the one who is proposing to replace existing code. Before the commons-number solution could be integrated into commons-imaging, someone would have to inspect both approaches method-by-method to verify that the results are compatible. In doing so, he might discover that the JUnit tests in commons-imaging needed to be enhanced to pick up the edge cases (after all, imaging is a pretty old set of code that was written before JUnit tests had come into wide practice). The idea of promoting code reuse is usually a good idea. But in Imaging-356, we saw recently that changing legacy code to use an alternate API can go badly wrong. Now, regarding the idea of bloat, we will never agree. And I maintain that I have a point that cannot be dismissed out-of-hand. Commons-numbers-fraction is only 25.7 K with 7 compiled classes. That's no big deal in a modern computing environment. But, oh yeah, it depends on commons-number-core (25,3 K, 14 compiled classes). So now my Configuration Management team, to whom I am answerable, has to add two new Jar files to their build. And adding a dependency on commons-numbers will be adding multiple classes that are unrelated to what commons-imaging does. We use a lot of commons APIs, but we won't currently use the numbers classes. Finally, on a less contentious topic, I did want to add that the refactoring of the Math project into smaller pieces is an excellent idea and that you guys have made an outstanding effort. Some years back, I went through the mental exercise of wondering what would be involved in reorganizing math into separate modules (a core including linear algebra? where would combinatorics go? can you separate statistics from differential equations? how would you handle complex numbers?). I quickly concluded that it would be a tricky job with a lot of hard decisions and that I wasn't even going to propose it. So I do admire what you were able to accomplish. was (Author: gwlucas): So what you are proposing is to replace code that has been around for long time with an alternate module. I cited concerns that the alternate approach might not work due to the possibility of difference in implementation and rules. You asked the reasonable question "where are the differences?" And now I will make the reasonable observation that the onus is on you to answer that, not me. After all, you are the one who is proposing to replace existing code. Before the commons-number solution could be integrated into commons-imaging, someone would have to inspect both approaches method-by-method to verify that the results are compatible. In doing so, he might discover that the JUnit tests in commons-imaging needed to be enhanced to pick up the edge cases (after all, imaging is a pretty old set of code that was written before JUnit tests had come into wide practice). The idea of promoting code reuse is usually a good idea. But in Imaging-356, we saw recently that changing legacy code to use an alternate API's can go badly wrong. Now, regarding the idea of bloat, we will never agree. And I maintain that I have a point that cannot be dismissed out-of-hand. Commons-numbers-fraction is only 25.7 K with 7 compiled classes. That's no big deal in a modern computing environment. But, oh yeah, it depends on commons-number-core (25,3 K, 14 compiled classes). So now my Configuration Management team, to whom I am answerable, has to add two new Jar files to their build. And adding a dependency on commons-numbers will be adding multiple classes that are unrelated to what commons-imaging does. We use a lot of commons APIs, but we won't currently use the numbers classes. Finally, on a less contentious topic, I did want to add that the refactoring of the Math project into smaller pieces is an excellent idea and that you guys have made an outstanding effort. Some years back, I went through the mental exercise of wondering what would be involved in reorganizing math into separate modules (a core including linear algebra? where would combinatorics go? can you separate statistics from differential equations? how would you handle complex numbers?). I quickly concluded that it would be a tricky job with a lot
[jira] [Commented] (IMAGING-358) "RationalNumber" class is "public"
[
https://issues.apache.org/jira/browse/IMAGING-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743803#comment-17743803
]
Gary Lucas commented on IMAGING-358:
So what you are proposing is to replace code that has been around for long
time with an alternate module. I cited concerns that the alternate approach
might not work due to the possibility of difference in implementation and rule.
You asked the reasonable question "where are the differences?" And now I
will make the reasonable observation that the onus is on you to answer that,
not me. After all, you are the one who is proposing to replace existing code.
Before the commons-number solution could be integrated into commons-imaging,
someone would have to inspect both approaches method-by-method to verify that
the results are compatible. In doing so, he might discover that the JUnit
tests in commons-imaging needed to be enhanced to pick up the edge cases (after
all, imaging is a pretty old set of code that was written before JUnit tests
had come into wide practice).
The idea of promoting code reuse is usually a good idea. But in Imaging-356,
we saw recently that changing legacy code to use an alternate API's can go
badly wrong.
Now, regarding the idea of bloat, we will never agree. And I maintain that I
have a point that cannot be dismissed out-of-hand. Commons-numbers-fraction is
only 25.7 K with 7 compiled classes. That's no big deal in a modern computing
environment. But, oh yeah, it depends on commons-number-core (25,3 K, 14
compiled classes). So now my Configuration Management team, to whom I am
answerable, has to add two new Jar files to their build. And adding a
dependency on commons-numbers will be adding multiple classes that are
unrelated to what commons-imaging does. We use a lot of commons APIs, but we
won't currently use the numbers classes.
Finally, on a less contentious topic, I did want to add that the refactoring of
the Math project into smaller pieces is an excellent idea and that you guys
have made an outstanding effort. Some years back, I went through the mental
exercise of wondering what would be involved in reorganizing math into separate
modules (a core including linear algebra? where would combinatorics go? can you
separate statistics from differential equations? how would you handle complex
numbers?). I quickly concluded that it would be a tricky job with a lot of
hard decisions and that I wasn't even going to propose it. So I do admire
what you were able to accomplish.
> "RationalNumber" class is "public"
> --
>
> Key: IMAGING-358
> URL: https://issues.apache.org/jira/browse/IMAGING-358
> Project: Commons Imaging
> Issue Type: Wish
> Components: imaging.common.*
>Affects Versions: 1.0-alpha2
>Reporter: Gilles Sadowski
>Priority: Major
> Labels: API, support
> Fix For: 1.0, 1.0-alpha3
>
>
> As per its Javadoc, class
> [{{RationalNumber}}|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/common/RationalNumber.html]
> is tuned to the requirements of the TIFF format while its name purports that
> it's a general implementation of the "fraction" concept.
> Which is it?
> Do we want that utility to be part of [Imaging]'s public API (thus eliciting
> support to application developers who might use it beyond its intended scope)?
> A dependency on [[Numbers]'s "fraction"
> module|https://commons.apache.org/proper/commons-numbers/commons-numbers-docs/apidocs/org/apache/commons/numbers/fraction/package-summary.html],
> as proposed in IMAGING-309) would avoid the issue, and also consolidate
> (and/or help find potential bugs in) [Numbers]'s implementation.
> If that proposal is not accepted, {{RationalNumber}} should preferably be
> moved to [Imaging]'s [{{internal}}
> package|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/internal/package-summary.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (IMAGING-358) "RationalNumber" class is "public"
[ https://issues.apache.org/jira/browse/IMAGING-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743803#comment-17743803 ] Gary Lucas edited comment on IMAGING-358 at 7/17/23 1:16 PM: - So what you are proposing is to replace code that has been around for long time with an alternate module. I cited concerns that the alternate approach might not work due to the possibility of difference in implementation and rules. You asked the reasonable question "where are the differences?" And now I will make the reasonable observation that the onus is on you to answer that, not me. After all, you are the one who is proposing to replace existing code. Before the commons-number solution could be integrated into commons-imaging, someone would have to inspect both approaches method-by-method to verify that the results are compatible. In doing so, he might discover that the JUnit tests in commons-imaging needed to be enhanced to pick up the edge cases (after all, imaging is a pretty old set of code that was written before JUnit tests had come into wide practice). The idea of promoting code reuse is usually a good idea. But in Imaging-356, we saw recently that changing legacy code to use an alternate API's can go badly wrong. Now, regarding the idea of bloat, we will never agree. And I maintain that I have a point that cannot be dismissed out-of-hand. Commons-numbers-fraction is only 25.7 K with 7 compiled classes. That's no big deal in a modern computing environment. But, oh yeah, it depends on commons-number-core (25,3 K, 14 compiled classes). So now my Configuration Management team, to whom I am answerable, has to add two new Jar files to their build. And adding a dependency on commons-numbers will be adding multiple classes that are unrelated to what commons-imaging does. We use a lot of commons APIs, but we won't currently use the numbers classes. Finally, on a less contentious topic, I did want to add that the refactoring of the Math project into smaller pieces is an excellent idea and that you guys have made an outstanding effort. Some years back, I went through the mental exercise of wondering what would be involved in reorganizing math into separate modules (a core including linear algebra? where would combinatorics go? can you separate statistics from differential equations? how would you handle complex numbers?). I quickly concluded that it would be a tricky job with a lot of hard decisions and that I wasn't even going to propose it. So I do admire what you were able to accomplish. was (Author: gwlucas): So what you are proposing is to replace code that has been around for long time with an alternate module. I cited concerns that the alternate approach might not work due to the possibility of difference in implementation and rule. You asked the reasonable question "where are the differences?" And now I will make the reasonable observation that the onus is on you to answer that, not me. After all, you are the one who is proposing to replace existing code. Before the commons-number solution could be integrated into commons-imaging, someone would have to inspect both approaches method-by-method to verify that the results are compatible. In doing so, he might discover that the JUnit tests in commons-imaging needed to be enhanced to pick up the edge cases (after all, imaging is a pretty old set of code that was written before JUnit tests had come into wide practice). The idea of promoting code reuse is usually a good idea. But in Imaging-356, we saw recently that changing legacy code to use an alternate API's can go badly wrong. Now, regarding the idea of bloat, we will never agree. And I maintain that I have a point that cannot be dismissed out-of-hand. Commons-numbers-fraction is only 25.7 K with 7 compiled classes. That's no big deal in a modern computing environment. But, oh yeah, it depends on commons-number-core (25,3 K, 14 compiled classes). So now my Configuration Management team, to whom I am answerable, has to add two new Jar files to their build. And adding a dependency on commons-numbers will be adding multiple classes that are unrelated to what commons-imaging does. We use a lot of commons APIs, but we won't currently use the numbers classes. Finally, on a less contentious topic, I did want to add that the refactoring of the Math project into smaller pieces is an excellent idea and that you guys have made an outstanding effort. Some years back, I went through the mental exercise of wondering what would be involved in reorganizing math into separate modules (a core including linear algebra? where would combinatorics go? can you separate statistics from differential equations? how would you handle complex numbers?). I quickly concluded that it would be a tricky job with a lot
[jira] [Commented] (IMAGING-358) "RationalNumber" class is "public"
[
https://issues.apache.org/jira/browse/IMAGING-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743590#comment-17743590
]
Gary Lucas commented on IMAGING-358:
It's been two years since I looked at this (Imaging-285), but as I recall the
differences between the implementations in Commons Imaging and Commons Numbers
were related to the way the two classes treated 32-bit unsigned integers
(which, of course, don't exist in Java). So if we were going to consider adding
a dependency on Commons Numbers, we would have to look at the problem very
carefully. While I believe that I saw potential problems back when I looked at
Imaging-309, I do not recall what they were. So I could be wrong. The idea
might work. Of course, it might work only if we added special-case
modifications to Commons Numbers. If that were the case, then I don't think it
would be appropriate to modify the more general purpose library (Numbers) to
suit a special-case requirement for the more narrow application (Imaging).
Now, in answer to the question "where is the bloat?", I think the question is a
bit disingenuous. Commons Imaging started with no external dependencies except
the Java standard API. If we were to add a dependency to another library, and
then that library added dependencies of its own, we would have bloat. You may
disagree with my reluctance to add more external dependencies. You may even be
right. But do not pretend that doing so does not carry a cost.
> "RationalNumber" class is "public"
> --
>
> Key: IMAGING-358
> URL: https://issues.apache.org/jira/browse/IMAGING-358
> Project: Commons Imaging
> Issue Type: Wish
> Components: imaging.common.*
>Affects Versions: 1.0-alpha2
>Reporter: Gilles Sadowski
>Priority: Major
> Labels: API, support
> Fix For: 1.0, 1.0-alpha3
>
>
> As per its Javadoc, class
> [{{RationalNumber}}|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/common/RationalNumber.html]
> is tuned to the requirements of the TIFF format while its name purports that
> it's a general implementation of the "fraction" concept.
> Which is it?
> Do we want that utility to be part of [Imaging]'s public API (thus eliciting
> support to application developers who might use it beyond its intended scope)?
> A dependency on [[Numbers]'s "fraction"
> module|https://commons.apache.org/proper/commons-numbers/commons-numbers-docs/apidocs/org/apache/commons/numbers/fraction/package-summary.html],
> as proposed in IMAGING-309) would avoid the issue, and also consolidate
> (and/or help find potential bugs in) [Numbers]'s implementation.
> If that proposal is not accepted, {{RationalNumber}} should preferably be
> moved to [Imaging]'s [{{internal}}
> package|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/internal/package-summary.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (IMAGING-358) "RationalNumber" class is "public"
[
https://issues.apache.org/jira/browse/IMAGING-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743563#comment-17743563
]
Gary Lucas edited comment on IMAGING-358 at 7/16/23 8:11 PM:
-
Well, there always a trade off with this kind of thing. In general, I really
dislike the idea of adding new dependencies to Commons Imaging (or any other
software package). As a software developer, it makes my life more complicated
when the library I want to use requires that I also include others I don't
need. However, if there is a module that is truly common, then it doesn't make
sense to implement it different ways. Adding unnecessary libraries adds bloat.
But implementing redundant classes also adds bloat. Is there a Nash balance
somewhere in this game? Finding it might be a challenge.
In the case of Imaging-309, it turned out that the rational number format used
by TIFF just wasn't a perfect fit with that used in the math library. Some of
the rules were a little different. And since the rules were defined as part of
the TIFF format specification, it wasn't like we could change them. And
modifying the math libary's NumberFormat class struck me as a bit risky because
we would be saddling a generic implementation with some ad hoc requirements to
support a single edge case (the TIFF format).
Finally, I'm not sure about the scoping. As I recall, RationalNumber was used
in multiple packages. I will look at the code later.
was (Author: gwlucas):
Well, there always a trade off with this kind of thing. In general, I really
dislike the idea of adding new dependencies to Commons Imaging (or any other
software package). As a software developer, it makes my life more complicated
when the library I want to use requires that I also include others I don't
need. However, if there is a module that is truly common, then it doesn't make
sense to implement it different ways. Adding unnecessary libraries adds bloat.
But implementing redundant classes also adds bloat. Is there a Nash balance
somewhere in this game? Finding it might be a challenge.
In the case of Imaging-309, it turned out that the rational number format used
by TIFF just wasn't a perfect fit with that used in the math library. Some of
the rules were a little different. And since the rules were defined as part of
the TIFF format specification, it wasn't like we could change them. I'm not
sure about the scoping. As I recall, RationalNumber was used in multiple
packages. I will look at the code later.
> "RationalNumber" class is "public"
> --
>
> Key: IMAGING-358
> URL: https://issues.apache.org/jira/browse/IMAGING-358
> Project: Commons Imaging
> Issue Type: Wish
> Components: imaging.common.*
>Affects Versions: 1.0-alpha2
>Reporter: Gilles Sadowski
>Priority: Major
> Labels: API, support
> Fix For: 1.0, 1.0-alpha3
>
>
> As per its Javadoc, class
> [{{RationalNumber}}|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/common/RationalNumber.html]
> is tuned to the requirements of the TIFF format while its name purports that
> it's a general implementation of the "fraction" concept.
> Which is it?
> Do we want that utility to be part of [Imaging]'s public API (thus eliciting
> support to application developers who might use it beyond its intended scope)?
> A dependency on [[Numbers]'s "fraction"
> module|https://commons.apache.org/proper/commons-numbers/commons-numbers-docs/apidocs/org/apache/commons/numbers/fraction/package-summary.html],
> as proposed in IMAGING-309) would avoid the issue, and also consolidate
> (and/or help find potential bugs in) [Numbers]'s implementation.
> If that proposal is not accepted, {{RationalNumber}} should preferably be
> moved to [Imaging]'s [{{internal}}
> package|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/internal/package-summary.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-358) "RationalNumber" class is "public"
[
https://issues.apache.org/jira/browse/IMAGING-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743563#comment-17743563
]
Gary Lucas commented on IMAGING-358:
Well, there always a trade off with this kind of thing. In general, I really
dislike the idea of adding new dependencies to Commons Imaging (or any other
software package). As a software developer, it makes my life more complicated
when the library I want to use requires that I also include others I don't
need. However, if there is a module that is truly common, then it doesn't make
sense to implement it different ways. Adding unnecessary libraries adds bloat.
But implementing redundant classes also adds bloat. Is there a Nash balance
somewhere in this game? Finding it might be a challenge.
In the case of Imaging-309, it turned out that the rational number format used
by TIFF just wasn't a perfect fit with that used in the math library. Some of
the rules were a little different. And since the rules were defined as part of
the TIFF format specification, it wasn't like we could change them. I'm not
sure about the scoping. As I recall, RationalNumber was used in multiple
packages. I will look at the code later.
> "RationalNumber" class is "public"
> --
>
> Key: IMAGING-358
> URL: https://issues.apache.org/jira/browse/IMAGING-358
> Project: Commons Imaging
> Issue Type: Wish
> Components: imaging.common.*
>Affects Versions: 1.0-alpha2
>Reporter: Gilles Sadowski
>Priority: Major
> Labels: API, support
> Fix For: 1.0, 1.0-alpha3
>
>
> As per its Javadoc, class
> [{{RationalNumber}}|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/common/RationalNumber.html]
> is tuned to the requirements of the TIFF format while its name purports that
> it's a general implementation of the "fraction" concept.
> Which is it?
> Do we want that utility to be part of [Imaging]'s public API (thus eliciting
> support to application developers who might use it beyond its intended scope)?
> A dependency on [[Numbers]'s "fraction"
> module|https://commons.apache.org/proper/commons-numbers/commons-numbers-docs/apidocs/org/apache/commons/numbers/fraction/package-summary.html],
> as proposed in IMAGING-309) would avoid the issue, and also consolidate
> (and/or help find potential bugs in) [Numbers]'s implementation.
> If that proposal is not accepted, {{RationalNumber}} should preferably be
> moved to [Imaging]'s [{{internal}}
> package|https://commons.apache.org/proper/commons-imaging/apidocs/org/apache/commons/imaging/internal/package-summary.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739752#comment-17739752
]
Gary Lucas commented on IMAGING-356:
Nice job, it will be interesting to see what you did.
Running your new code on PICT2833.TIF, the speed and memory test yielded
{code:java}
time to load image-- memory
time ms avg ms --used mb total mb
[snip...]
22.93723.390-- 48.259 479.500
{code}
The "avg ms" is the average time in milliseconds to load the image. The way
"speed and memory" works is that it loads the image 10 times. The first three
times are ignored by the tabulation (the idea being that they include overhead
for the classloader and JIT compiler). The worst of the 10 trials is also
ignored. The first column gives the time for the current test, but it really
doesn't mean much (I included it only to see if something abnormal happens
during the test).
So that was the good news and it is an outstanding improvement. I am a bit
reluctant to mention it, but there might still be an opportunity for a small
improvement. Looking at the same test using the previous version of the code
(before Commons IO was integrated into it), I see the following results.
{code:java}
time to load image-- memory
time ms avg ms --used mb total mb
19.81520.424-- 33.896 479.500
{code}
So the previous version used less memory and took less time to operate. I
repeated the test several times. That being said, the previous version was
1.0-alpha3 which I downloaded and compiled 24 May 2022. Could the difference be
somehow related to changes in the pom.xml?
Also, I have not looked at other formats or even other TIFF files at this time.
I will do a bit more testing with some of the 300 MB aerial photograph files I
use and will let you know if I find anything noteworthy.
> TIFF reading extremely slow in version 1.0-SNAPSHOT
> ---
>
> Key: IMAGING-356
> URL: https://issues.apache.org/jira/browse/IMAGING-356
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Affects Versions: 1.0
>Reporter: Gary Lucas
>Priority: Major
>
> I am using the latest code from github (1.0-SNAPSHOT downloaded from github
> of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required
> 673 milliseconds to read that file. The new code requires upward of 15
> minutes. Clearly something got broken since the last release.
> The TIFF file is a 1x1 pixel 4 byte image format organized in strips.
> The bottleneck appears to occur in the TiffReader getTiffRawImageData method
> which reads raw data from the file in preparation of creating a BufferedImage
> object.
> I suspect that there may be a general slowness of file access. In debugging,
> even reading the initial metadata (22 TIFF tags) took a couple of seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739567#comment-17739567 ] Gary Lucas commented on IMAGING-356: Ha! Good one. What I meant to say, of course, is that it was in the package dedicated to the TIFF format... org.apache.commons.imaging.formats.tiff. and you can find that specific class under the "data readers" siubdirectory org.apache.commons.imaging.formats.tiff.datareaders > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739420#comment-17739420 ] Gary Lucas commented on IMAGING-356: Any good sized TIFF file would do (not sure about other formats). The one I used was located in https://github.com/apache/commons-imaging/tree/master/src/test/data/images/tiff/1/PICT2833.TIF I picked that one just because it was large enough that I could make meaningful access time measurements, but not so large that it would be impractical. Also, it was already part of the distribution, so it wouldn't add anything new to the project. Internally, it is accessed by the TIFF branch of Commons Imaging in a class called DataReaderStrips. Good luck. > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739377#comment-17739377 ] Gary Lucas commented on IMAGING-356: The glitch in the Allocator class is that there are a couple of non UTF-8 valid characters in the Javadoc. So it doesn't affect anything code-wise and doesn't urgently require attention. I might submit a PR at some point. The speed and memory test isn't a JUnit test. It's a tool for developers. It's remarkable how many "good ideas" about improving performance do no such thing, So it was useful to have an unambiguous benchmark to measure stuff. We were interested in looking at aerial photographs for geospatial analysis logic. I spent weeks trying to get the Imaging performance to the point where it was useful in an application. It was quite the learning experience Since then, I always encourage folks to do benchmarking as well as JUnit tests. > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739371#comment-17739371 ] Gary Lucas commented on IMAGING-356: No luck. Same runtime as before... Though the change did reduce the memory use quite a bit. First, to confirm that I tested the right thing... You pushed your changes directly to the master branch? That's where I downloaded code and I did see the commit comment saying "Delegate size". I also stepped through the code just to be sure that a change was in effect. The change I saw looks much more reasonable than the previous approach. Did you try running the ApacheImagingSpeedAndMemoryTest? I wrote it back when I was trying to optimize the speed of the imaging classes (the old Sanselan classes). I tried a lot of different ideas before I got things running at a reasonable speed. It was very helpful. For this test, I tried processing the file called PICT2833.TIF which is included in the project distribution as part of a suite of test files. Gary P.S. On an separate issue, have you been seeing complaints about encoding errors in the Allocator class? I am seeing invalid UTF-8 characters in the Javadoc comments on lines 144 and 165. > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739288#comment-17739288 ] Gary Lucas commented on IMAGING-356: I haven't studied the changes that were made, so I can't offer any authoritative recommendations on the approach. Instead, I have a few general observations about the way TIFF files work that may be useful in figuring how you tackle the problem. Or perhaps not. So take them with a grain of salt. TIFF files are kind of a special case in terms of image formats. First off, one can never assume that a TIFF file is going to be accessed in-order. It is common for the the "directory" section of the file (which tells how its organized) to come last rather than first. And, of course, a TIFF file may have multiple directories (because it may contain multiple images). Second, TIFF files are typically quite large, often in the hundreds of megabytes range, and sometimes in the gigabyte range. So it is often preferred to not keep the entire thing in memory. In many cases, an application will not access the entire file, but only a subsection. For example, a mapping program displaying an aerial photograph might only access the subsection of the photograph that is actually visible on the map. And finally, I note that TIFF files are often not images at all, but are used to store numerical raster data (such as Earth elevation and ocean depth data). All of this means that the file-access pattern for a TIFF file is a closer fit to the idea of a random access file rather than the idea of a sequential IO channel such as a network socket or a serial device. I know that the PNG format (the only other one I've studied in depth) was designed with network access specifically in mind. The TIFF format evolved before network access was in the ascendency as it is today. That being said, even the original Commons Imaging approach to TIFF file IO wasn't quite a perfect fit. For one thing, the original authors open and close a file multiple times (as they access each part of the file) . That is suboptimal since opening and closing a file carries its own performance overhead. Also, when I was looking at refactoring Commons Imaging IO to implement Closeable to support of try-with-resources blocks, I didn't see a way to accomplish that without a significant rewrite and compatibility breaking changes to the public API. > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739170#comment-17739170
]
Gary Lucas edited comment on IMAGING-356 at 6/30/23 4:57 PM:
-
Looking at the code history in github, I may have found one issue, though I am
not sure that it is the major issue. In the ByteSource class, there is a call
to a function called size with the following logic
{code:java}
public long size() throws IOException {
return origin.getByteArray().length;
}
{code}
So the question comes up, how much work is involved in a call to getByteArray?
It looks like "origin" is an object of type AbstractOrigin.FileOrigin() and the
call it makes to getByteArray is
{code:java}
@Override
public byte[] getByteArray() throws IOException {
return Files.readAllBytes(getPath());
}{code}
Which, of course, is a pretty expensive call. On the other hand, the size()
method is only called 12 times when loading the PICT2883.TIF image. But each
call does pull back 14788608 bytes.
I think that this may have occurred when commons.imaging was refactored to use
commons.io.
It may be that there's an impedance mismatch between the ideas from commons.io
and the assumptions in the commons.imaging classes.
was (Author: gwlucas):
Looking at the code history in github, I may have found one issue, though I am
not sure that it is the major issue. In the ByteSource class, there is a call
to a function called size with the following logic
{code:java}
public long size() throws IOException {
return origin.getByteArray().length;
}
{code}
So the question comes up, how much work is involved in a call to getByteArray?
It looks like "origin" is an object of type AbstractOrigin.FileOrigin() and the
call it makes to getByteArray is
{code:java}
@Override
public byte[] getByteArray() throws IOException {
return Files.readAllBytes(getPath());
}{code}
Which, of course, is a pretty expensive call. On the other hand, the size()
method is only called 12 times when loading the PICT2883.TIF image. But each
call does pull back 14788608 bytes.
I think that this my be a case where there's an impedance mismatch between the
ideas from commons.io and the assumptions in the commons.imaging classes.
> TIFF reading extremely slow in version 1.0-SNAPSHOT
> ---
>
> Key: IMAGING-356
> URL: https://issues.apache.org/jira/browse/IMAGING-356
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Affects Versions: 1.0
>Reporter: Gary Lucas
>Priority: Major
>
> I am using the latest code from github (1.0-SNAPSHOT downloaded from github
> of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required
> 673 milliseconds to read that file. The new code requires upward of 15
> minutes. Clearly something got broken since the last release.
> The TIFF file is a 1x1 pixel 4 byte image format organized in strips.
> The bottleneck appears to occur in the TiffReader getTiffRawImageData method
> which reads raw data from the file in preparation of creating a BufferedImage
> object.
> I suspect that there may be a general slowness of file access. In debugging,
> even reading the initial metadata (22 TIFF tags) took a couple of seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[
https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739170#comment-17739170
]
Gary Lucas commented on IMAGING-356:
Looking at the code history in github, I may have found one issue, though I am
not sure that it is the major issue. In the ByteSource class, there is a call
to a function called size with the following logic
{code:java}
public long size() throws IOException {
return origin.getByteArray().length;
}
{code}
So the question comes up, how much work is involved in a call to getByteArray?
It looks like "origin" is an object of type AbstractOrigin.FileOrigin() and the
call it makes to getByteArray is
{code:java}
@Override
public byte[] getByteArray() throws IOException {
return Files.readAllBytes(getPath());
}{code}
Which, of course, is a pretty expensive call. On the other hand, the size()
method is only called 12 times when loading the PICT2883.TIF image. But each
call does pull back 14788608 bytes.
I think that this my be a case where there's an impedance mismatch between the
ideas from commons.io and the assumptions in the commons.imaging classes.
> TIFF reading extremely slow in version 1.0-SNAPSHOT
> ---
>
> Key: IMAGING-356
> URL: https://issues.apache.org/jira/browse/IMAGING-356
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: TIFF
>Affects Versions: 1.0
>Reporter: Gary Lucas
>Priority: Major
>
> I am using the latest code from github (1.0-SNAPSHOT downloaded from github
> of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required
> 673 milliseconds to read that file. The new code requires upward of 15
> minutes. Clearly something got broken since the last release.
> The TIFF file is a 1x1 pixel 4 byte image format organized in strips.
> The bottleneck appears to occur in the TiffReader getTiffRawImageData method
> which reads raw data from the file in preparation of creating a BufferedImage
> object.
> I suspect that there may be a general slowness of file access. In debugging,
> even reading the initial metadata (22 TIFF tags) took a couple of seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739148#comment-17739148 ] Gary Lucas commented on IMAGING-356: In response to the comment "You should bisect to the offending change", I don't think that's really my responsibility. I'm not the one who made the code change that degraded performance. Even so, I did try to narrow down the issue and identify a test procedure that would help address the problem. We all make mistakes. I am hoping that, once alerted to the issue, whoever is working in this area will recognize his or her mistake and find it relatively easy to make a fix. > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/IMAGING-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739138#comment-17739138 ] Gary Lucas commented on IMAGING-356: This might help. It turns out that you don't need one of my big GeoTIFF files to test this feature. You can accomplish adequate testing with resources already included in the distribution. I ran tests using the ApacheImagingSpeedAndMemoryTest application that's included in the Commons Imaging source distribution. I used it to read the test file PICT2883.TIF which is also included. The speed and memory test reads a single file and instruments how long it takes to read Running the older Alpha 3 version: Avg time 18 milliseconds Running the latest code version: Avg time 1226 milliseconds The new version also seems to be consuming quite a bit more memory than the older version. > TIFF reading extremely slow in version 1.0-SNAPSHOT > --- > > Key: IMAGING-356 > URL: https://issues.apache.org/jira/browse/IMAGING-356 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0 >Reporter: Gary Lucas >Priority: Major > > I am using the latest code from github (1.0-SNAPSHOT downloaded from github > of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required > 673 milliseconds to read that file. The new code requires upward of 15 > minutes. Clearly something got broken since the last release. > The TIFF file is a 1x1 pixel 4 byte image format organized in strips. > The bottleneck appears to occur in the TiffReader getTiffRawImageData method > which reads raw data from the file in preparation of creating a BufferedImage > object. > I suspect that there may be a general slowness of file access. In debugging, > even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IMAGING-356) TIFF reading extremely slow in version 1.0-SNAPSHOT
Gary Lucas created IMAGING-356: -- Summary: TIFF reading extremely slow in version 1.0-SNAPSHOT Key: IMAGING-356 URL: https://issues.apache.org/jira/browse/IMAGING-356 Project: Commons Imaging Issue Type: Bug Components: Format: TIFF Affects Versions: 1.0 Reporter: Gary Lucas I am using the latest code from github (1.0-SNAPSHOT downloaded from github of June 2023) to read a 300 megabyte TIFF file. Version 1.0-alpha3 required 673 milliseconds to read that file. The new code requires upward of 15 minutes. Clearly something got broken since the last release. The TIFF file is a 1x1 pixel 4 byte image format organized in strips. The bottleneck appears to occur in the TiffReader getTiffRawImageData method which reads raw data from the file in preparation of creating a BufferedImage object. I suspect that there may be a general slowness of file access. In debugging, even reading the initial metadata (22 TIFF tags) took a couple of seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724579#comment-17724579
]
Gary Lucas commented on IMAGING-319:
On further review, it appears that the section of the code that looks for
best-fit is actually correct... It's just confusing. So I will be adding
comments to clarify what it does.
The reason that it works is that the unclaimed-memory elements are kept in
sorted order, from largest to smallest.
However, the logic related to the element length is still wrong. I am
implementing a fix and will activate thr JUnit test ExifRewriterRoundtripTest
once I get things working.
> updateExifMetadataLossless lost the first character of a String
> ---
>
> Key: IMAGING-319
> URL: https://issues.apache.org/jira/browse/IMAGING-319
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: JPEG
>Affects Versions: 1.0-alpha2
>Reporter: Sicheng Yang
>Assignee: Bruno P. Kinoshita
>Priority: Major
> Attachments: Screen Shot 2021-11-26 at 4.01.06 PM-1.png, Screen Shot
> 2021-11-26 at 4.01.21 PM-1.png, iPhone12-geotag.JPG
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I try to use TiffOutputSet to generate a new image. However, if a tag that
> contains String, the program may miss the first character of the String.
>
> import java.io.*;
> import org.apache.commons.imaging.ImageReadException;
> import org.apache.commons.imaging.ImageWriteException;
> import org.apache.commons.imaging.Imaging;
> import org.apache.commons.imaging.common.ImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.JpegImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.exif.ExifRewriter;
> import org.apache.commons.imaging.formats.tiff.TiffImageMetadata;
> import org.apache.commons.imaging.formats.tiff.write.TiffOutputSet;
> public class LibraryTest {
> public static void main(String[] args) throws ImageReadException,
> IOException, ImageWriteException {
> File source = new File("./assets/iPhone12-geotag.JPG");
> File result = new
> File("./assets/results/editted-iPhone12-geotag.JPG");
> final ImageMetadata metadata = Imaging.getMetadata(source);
> final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
> final TiffImageMetadata exif = jpegMetadata.getExif();
> TiffOutputSet outputSet = exif.getOutputSet();
> BufferedOutputStream bufferedOutputStream = new
> BufferedOutputStream(new FileOutputStream(result));
> new ExifRewriter().updateExifMetadataLossless(source,
> bufferedOutputStream, outputSet);
> }
> }
>
> This is the sample code.
> Tag value in original image
> !image-2021-11-26-16-01-58-645.png!
> Tag value in output image
> !image-2021-11-26-16-04-12-185.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724572#comment-17724572
]
Gary Lucas commented on IMAGING-319:
I am looking at this now. One thing I've noticed is that the code contains a
block described as "search for the smallest possible element large enough to
hold the item". But it is actually a "first-fit" approach rather than a
"best-fit" approach. Does anyone see a motivation for refactoring this to
actually use a best-fit criteria? Or should I just correct the comments? I
seem to recall, from a long time ago, reading an article that claimed that
malloc() uses a first-fit approach because best-fit leads to fragmented
memory... but I don't recall any kind of proof or analysis for that claim.
Here's the code in question. It starts at line 194 of
TiffImageWriterLossless.java
{code:java}
final TiffOutputItem outputItem = unplacedItems.remove(0);
final int outputItemLength = outputItem.getItemLength();
// search for the smallest possible element large enough to hold the
// item.
TiffElement bestFit = null;
for (final TiffElement element : unusedElements) {
if (element.length < outputItemLength) {
break;
}
bestFit = element;
}
{code}
> updateExifMetadataLossless lost the first character of a String
> ---
>
> Key: IMAGING-319
> URL: https://issues.apache.org/jira/browse/IMAGING-319
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: JPEG
>Affects Versions: 1.0-alpha2
>Reporter: Sicheng Yang
>Assignee: Bruno P. Kinoshita
>Priority: Major
> Attachments: Screen Shot 2021-11-26 at 4.01.06 PM-1.png, Screen Shot
> 2021-11-26 at 4.01.21 PM-1.png, iPhone12-geotag.JPG
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I try to use TiffOutputSet to generate a new image. However, if a tag that
> contains String, the program may miss the first character of the String.
>
> import java.io.*;
> import org.apache.commons.imaging.ImageReadException;
> import org.apache.commons.imaging.ImageWriteException;
> import org.apache.commons.imaging.Imaging;
> import org.apache.commons.imaging.common.ImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.JpegImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.exif.ExifRewriter;
> import org.apache.commons.imaging.formats.tiff.TiffImageMetadata;
> import org.apache.commons.imaging.formats.tiff.write.TiffOutputSet;
> public class LibraryTest {
> public static void main(String[] args) throws ImageReadException,
> IOException, ImageWriteException {
> File source = new File("./assets/iPhone12-geotag.JPG");
> File result = new
> File("./assets/results/editted-iPhone12-geotag.JPG");
> final ImageMetadata metadata = Imaging.getMetadata(source);
> final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
> final TiffImageMetadata exif = jpegMetadata.getExif();
> TiffOutputSet outputSet = exif.getOutputSet();
> BufferedOutputStream bufferedOutputStream = new
> BufferedOutputStream(new FileOutputStream(result));
> new ExifRewriter().updateExifMetadataLossless(source,
> bufferedOutputStream, outputSet);
> }
> }
>
> This is the sample code.
> Tag value in original image
> !image-2021-11-26-16-01-58-645.png!
> Tag value in output image
> !image-2021-11-26-16-04-12-185.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720904#comment-17720904
]
Gary Lucas commented on IMAGING-319:
I will take a look. I've been away from Commons Imaging for awhile, so it's
going to take some time to get set up again.
As I recall, the code fix worked for the sample submitted by the original
author. Perhaps something has shifted in the code base since then. I'll also
look at pull 275.
If you have any recommendations, please let me know.
> updateExifMetadataLossless lost the first character of a String
> ---
>
> Key: IMAGING-319
> URL: https://issues.apache.org/jira/browse/IMAGING-319
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: JPEG
>Affects Versions: 1.0-alpha2
>Reporter: Sicheng Yang
>Assignee: Bruno P. Kinoshita
>Priority: Major
> Attachments: Screen Shot 2021-11-26 at 4.01.06 PM-1.png, Screen Shot
> 2021-11-26 at 4.01.21 PM-1.png, iPhone12-geotag.JPG
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I try to use TiffOutputSet to generate a new image. However, if a tag that
> contains String, the program may miss the first character of the String.
>
> import java.io.*;
> import org.apache.commons.imaging.ImageReadException;
> import org.apache.commons.imaging.ImageWriteException;
> import org.apache.commons.imaging.Imaging;
> import org.apache.commons.imaging.common.ImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.JpegImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.exif.ExifRewriter;
> import org.apache.commons.imaging.formats.tiff.TiffImageMetadata;
> import org.apache.commons.imaging.formats.tiff.write.TiffOutputSet;
> public class LibraryTest {
> public static void main(String[] args) throws ImageReadException,
> IOException, ImageWriteException {
> File source = new File("./assets/iPhone12-geotag.JPG");
> File result = new
> File("./assets/results/editted-iPhone12-geotag.JPG");
> final ImageMetadata metadata = Imaging.getMetadata(source);
> final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
> final TiffImageMetadata exif = jpegMetadata.getExif();
> TiffOutputSet outputSet = exif.getOutputSet();
> BufferedOutputStream bufferedOutputStream = new
> BufferedOutputStream(new FileOutputStream(result));
> new ExifRewriter().updateExifMetadataLossless(source,
> bufferedOutputStream, outputSet);
> }
> }
>
> This is the sample code.
> Tag value in original image
> !image-2021-11-26-16-01-58-645.png!
> Tag value in output image
> !image-2021-11-26-16-04-12-185.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IMAGING-330) Implement PNG predictors to reduce output size
[ https://issues.apache.org/jira/browse/IMAGING-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-330: --- Description: I propose to enhance the PngWriter class and PngImagingParameters class to allow the use of predictors. This change should reduce the size of output images written in PNG format. The resulting images will carry exactly the same data. There will be no loss of pixels or image quality. But the results will be smaller than those currently produced by either Imaging or Java's ImageIO class. Background The PNG specification permits the use of optional predictors as part of its data compression logic. Predictors are applied through the use of a filter that transforms the data before it is passed to the conventional Deflate data compressor. In some cases, predictors can result in a 30 percent reduction of file size. They are particularly suited to photographic images. Although they will work on graphics and line art, the reduction is often more modest. You can find a description of predictors on [Wikipedia|https://en.wikipedia.org/wiki/Portable_Network_Graphics#Filtering] The Java ImageIO class does not apply predictors as part of its processing. Consequently, if you write an image from a Java application using ImageIO, pull the image into Paint, and then save it under another name, the size of the image may actually decrease. So when this feature is added to Commons Imaging, it will out perform ImageIO when writing PNGs. > Implement PNG predictors to reduce output size > -- > > Key: IMAGING-330 > URL: https://issues.apache.org/jira/browse/IMAGING-330 > Project: Commons Imaging > Issue Type: Improvement > Components: Format: PNG >Reporter: Gary Lucas >Priority: Major > > I propose to enhance the PngWriter class and PngImagingParameters class to > allow the use of predictors. This change should reduce the size of output > images written in PNG format. The resulting images will carry exactly the > same data. There will be no loss of pixels or image quality. But the results > will be smaller than those currently produced by either Imaging or Java's > ImageIO class. > Background > The PNG specification permits the use of optional predictors as part of its > data compression logic. Predictors are applied through the use of a filter > that transforms the data before it is passed to the conventional Deflate data > compressor. In some cases, predictors can result in a 30 percent reduction > of file size. They are particularly suited to photographic images. Although > they will work on graphics and line art, the reduction is often more modest. > You can find a description of predictors on > [Wikipedia|https://en.wikipedia.org/wiki/Portable_Network_Graphics#Filtering] > The Java ImageIO class does not apply predictors as part of its processing. > Consequently, if you write an image from a Java application using ImageIO, > pull the image into Paint, and then save it under another name, the size of > the image may actually decrease. So when this feature is added to Commons > Imaging, it will out perform ImageIO when writing PNGs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IMAGING-330) Implement PNG predictors to reduce output size
Gary Lucas created IMAGING-330: -- Summary: Implement PNG predictors to reduce output size Key: IMAGING-330 URL: https://issues.apache.org/jira/browse/IMAGING-330 Project: Commons Imaging Issue Type: Improvement Components: Format: PNG Reporter: Gary Lucas -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IMAGING-329) Opportunity to enhance speed of PNG Read
Gary Lucas created IMAGING-329:
--
Summary: Opportunity to enhance speed of PNG Read
Key: IMAGING-329
URL: https://issues.apache.org/jira/browse/IMAGING-329
Project: Commons Imaging
Issue Type: Improvement
Components: Format: PNG
Environment:
Reporter: Gary Lucas
When reading an image file, the PNG reader makes calls to
BufferedImage.setRGB() for each pixel to be set in its output image. The
setRGB method has a lot of overhead, and we could speed up processing by
calling setRGB on an entire row of pixels rather than one-at-a-time.
The expediter loop also makes calls to it's own getRGB method which is generic
across all the different PNG formats (32-bit true color, 24-bit true-color,
grayscale, indexed color). This action involves a lot of conditional
evaluation, switch statements across formats. If we were to implement specific
loops for the most common formats (24 and 32 bit true color), we could
streamline reading for those formats.
I experimented with both of these approaches using a 5000-by-5000 RGB image
(24-bit true color without transparency or gamma correction). The results were:
Current Version 1-Alpha 3: 0.917 seconds
Set entire row of pixels: 0.717 seconds
Custom loop for format: 0.609 seconds
I note that the saving is not spectacular (it would have been more important a
decade ago when computers were slower), particularly since images of such a
large size don't usually use PNG as a data format.
{code:java}
for (int y = 0; y < height; y++) {
final byte[] unfiltered = getNextScanline(
is, pixelBytesPerScanLine, prev, bytesPerPixel);
prev = unfiltered;
final BitParser bitParser = new BitParser(
unfiltered, bitsPerPixel, bitDepth);
for (int x = 0; x < width; x++) {
final int rgb = getRGB(bitParser, x);
bi.setRGB(x, y, rgb);
}
}
{code}
And here's the modified inner loop for writing one row at a time:
{code:java}
final int []argb = new int[width];
for (int x = 0; x < width; x++) {
argb[x] = getRGB(bitParser, x); // from ScanExpediterSimple
}
bi.setRGB(0, y, width, 1, argb, 0, width);
{code}
And, finally, here's a modified block that avoids the getRGB method and just
processes bytes directly. It has to check on a couple of pre-conditions to
see if the data is in one of the frequently used formats, and then processes
the bytes from the source file without any of the bit-access methods used by
the ScanExpediterSimple class's getRGB method.
{code:java}
if (pngColorType == PngColorType.TRUE_COLOR
&& transparencyFilter == null
&& gammaCorrection == null)
{
for (int y = 0; y < height; y++) {
final byte[] unfiltered = getNextScanline(
is, pixelBytesPerScanLine, prev, bytesPerPixel);
int k = 0;
final int []argb = new int[width];
for (int x = 0; x < width; x++) {
int r = unfiltered[k++]&0xff;
int g = unfiltered[k++]&0xff;
int b = unfiltered[k++]&0xff;
argb[x] = 0xff00 | (r<<16)|(g<<8)|b;
}
bi.setRGB(0, y, width, 1, argb, 0, width);
}
return;
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491005#comment-17491005
]
Gary Lucas edited comment on IMAGING-319 at 2/11/22, 4:09 PM:
--
Okay, found it.
In the code below, the method looped through all the available free elements
and found one it calls "bestFit". It is going to store the new data into the
available space. But TIFF files have a rule that the offsets have to be an
even multiple of 2. So there's a check to see if the offset is odd and, if it
is, the code advances the offset forward one. The problem is that it doesn't
recognize that by advancing the offset, it's reduced the amount of available
space (bestFit.length). So, the "excessLength" computation below will be
incorrect. If some subsequent element is an exact match for the incorrect
excessLength value, it will overwrite the unused space and clobber whatever
follows. In this case, the thing that got clobbered was the first byte of
EXIF tag 0x9010.
The probability of this happening is small, but non-zero. It is just luck that
Sicheng Yang's data sample triggered the issue.
{code:java}
long offset = bestFit.offset;
if ((offset & 1L) != 0) {
offset += 1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (bestFit.length > outputItemLength) {
// not a perfect fit.
final long excessOffset = bestFit.offset + outputItemLength;
final int excessLength = bestFit.length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
{code}
I re-wrote the code as follows. It works. Writing a JUnit test for this is
going to be extremely difficult unless we want to include the large test file
in our code distribution (which I think would be a bad idea).
{code:java}
long offset = bestFit.offset;
int length = bestFit.length;
if ((offset & 1L) != 0) {
// offsets have to be at a multiple of 2
offset += 1;
length -=1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (length > outputItemLength) {
// not a perfect fit.
final long excessOffset = offset + outputItemLength;
final int excessLength = length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
{code}
was (Author: gwlucas):
Okay, found it.
In the code below, the method looped through all the available free elements
and found one it calls "bestFit". It is going to store the new data into the
available space. But TIFF files have a rule that the offsets have to be an
even multiple of 2. So there's a check to see if the offset is odd and, if it
is, the code advances the offset forward one. The problem is that it doesn't
recognize that by advancing the offset, it's reduced the amount of available
space (bestFit.length). So, the "excessLength" computation below will be
incorrect. If some subsequent element is an exact match for the incorrect
excessLength value, it will overwrite the unused space and clobber whatever
follows. In this case, the thing that got clobbered was the first byte of
EXIF tag 0x9010.
The probability of this happening is small, but non zero. It is just luck that
Sicheng Yang's data sample triggered the issue.
{code:java}
long offset = bestFit.offset;
if ((offset & 1L) != 0) {
offset += 1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (bestFit.length > outputItemLength) {
// not a perfect fit.
final long excessOffset = bestFit.offset + outputItemLength;
final int excessLength = bestFit.length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
[jira] [Comment Edited] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490983#comment-17490983
]
Gary Lucas edited comment on IMAGING-319 at 2/11/22, 4:09 PM:
--
I haven't figured this out yet, but I've narrowed down the cause to
TiffImageWriterLossless. There is a method that attempts to update the file
positions (offsets) of the various EXIF tags. It's called
updateOffsetsSteps().
The code is very confusing. As far as I can tell, at some point some of the
space in the output file is determined to be "unused" and updateOffsetSteps()
attempts to reuse it by finding available space and setting the tag output
position to the available space. If you bypass this operation by adding a
diagnostic call to unusedElements.clear() right after the unusedElements list
is established, everything works fine.
was (Author: gwlucas):
I haven't figured this out yet, but I've narrowed down the cause to
TiffImageWriterLossless. There is a method that attempts to update the file
positions (offsets) of the various EXIF tags. It's called
updateOffsetsSteps().
The code is very confusing. As far as I can tell, at some point some of the
space in the output file is determined to be "unused" and updateOffsetSteps()
attempts to reuse it by finding available space and setting the tag output
position to the available space. If you bypass this operation by adding a
diagnostic call to unusedElements.clear() right after the unusedElements list
is established, everything works fine.
The call stack is basically
ExifRewriter.updateExifMetadataLossles
ExifRewriter.writeExifSegment
TiffImageWriterLossless.write
TiffImageWriterLossless.updateOffsetsStep
> updateExifMetadataLossless lost the first character of a String
> ---
>
> Key: IMAGING-319
> URL: https://issues.apache.org/jira/browse/IMAGING-319
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: JPEG
>Affects Versions: 1.0-alpha2
>Reporter: Sicheng Yang
>Priority: Major
> Attachments: Screen Shot 2021-11-26 at 4.01.06 PM-1.png, Screen Shot
> 2021-11-26 at 4.01.21 PM-1.png, iPhone12-geotag.JPG
>
>
> I try to use TiffOutputSet to generate a new image. However, if a tag that
> contains String, the program may miss the first character of the String.
>
> import java.io.*;
> import org.apache.commons.imaging.ImageReadException;
> import org.apache.commons.imaging.ImageWriteException;
> import org.apache.commons.imaging.Imaging;
> import org.apache.commons.imaging.common.ImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.JpegImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.exif.ExifRewriter;
> import org.apache.commons.imaging.formats.tiff.TiffImageMetadata;
> import org.apache.commons.imaging.formats.tiff.write.TiffOutputSet;
> public class LibraryTest {
> public static void main(String[] args) throws ImageReadException,
> IOException, ImageWriteException {
> File source = new File("./assets/iPhone12-geotag.JPG");
> File result = new
> File("./assets/results/editted-iPhone12-geotag.JPG");
> final ImageMetadata metadata = Imaging.getMetadata(source);
> final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
> final TiffImageMetadata exif = jpegMetadata.getExif();
> TiffOutputSet outputSet = exif.getOutputSet();
> BufferedOutputStream bufferedOutputStream = new
> BufferedOutputStream(new FileOutputStream(result));
> new ExifRewriter().updateExifMetadataLossless(source,
> bufferedOutputStream, outputSet);
> }
> }
>
> This is the sample code.
> Tag value in original image
> !image-2021-11-26-16-01-58-645.png!
> Tag value in output image
> !image-2021-11-26-16-04-12-185.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491005#comment-17491005
]
Gary Lucas edited comment on IMAGING-319 at 2/11/22, 4:07 PM:
--
Okay, found it.
In the code below, the method looped through all the available free elements
and found one it calls "bestFit". It is going to store the new data into the
available space. But TIFF files have a rule that the offsets have to be an
even multiple of 2. So there's a check to see if the offset is odd and, if it
is, the code advances the offset forward one. The problem is that it doesn't
recognize that by advancing the offset, it's reduced the amount of available
space (bestFit.length). So, the "excessLength" computation below will be
incorrect. If some subsequent element is an exact match for the incorrect
excessLength value, it will overwrite the unused space and clobber whatever
follows. In this case, the thing that got clobbered was the first byte of
EXIF tag 0x9010.
The probability of this happening is small, but non zero. It is just luck that
Sicheng Yang's data sample triggered the issue.
{code:java}
long offset = bestFit.offset;
if ((offset & 1L) != 0) {
offset += 1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (bestFit.length > outputItemLength) {
// not a perfect fit.
final long excessOffset = bestFit.offset + outputItemLength;
final int excessLength = bestFit.length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
{code}
I re-wrote the code as follows. It works. Writing a JUnit test for this is
going to be extremely difficult unless we want to include the large test file
in our code distribution (which I think would be a bad idea).
{code:java}
long offset = bestFit.offset;
int length = bestFit.length;
if ((offset & 1L) != 0) {
// offsets have to be at a multiple of 2
offset += 1;
length -=1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (length > outputItemLength) {
// not a perfect fit.
final long excessOffset = offset + outputItemLength;
final int excessLength = length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
{code}
was (Author: gwlucas):
Okay, found it.
In the code below, the method looped through all the available free elements
and found one it calls "bestFit". It is going to store the new data into the
available space. But TIFF files have a rule that the offsets have to be an
even multiple of 2. So there's a check to see if the offset is odd and, if it
is, the code advances the offset forward one. The problem is that it doesn't
recognize that by advancing the offset, it's reduced the amount of available
space (bestFit.length). So, the "excessLength" computation below will be
incorrect. If some subsequent element is an exact match for the incorrect
excessLength value, it will overwrite the unused space and clobber whatever
follows. In this case, the thing that got clobbered was the first byte of
EXIF tag 0x9010.
The probability of this happening is small, but non zero. It is just luck that
Sicheng Yang's data sample triggered the issue.
long offset = bestFit.offset;
if ((offset & 1L) != 0) {
offset += 1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (bestFit.length > outputItemLength) {
// not a perfect fit.
final long excessOffset = bestFit.offset + outputItemLength;
final int excessLength = bestFit.length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
[jira] [Comment Edited] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491005#comment-17491005
]
Gary Lucas edited comment on IMAGING-319 at 2/11/22, 4:06 PM:
--
Okay, found it.
In the code below, the method looped through all the available free elements
and found one it calls "bestFit". It is going to store the new data into the
available space. But TIFF files have a rule that the offsets have to be an
even multiple of 2. So there's a check to see if the offset is odd and, if it
is, the code advances the offset forward one. The problem is that it doesn't
recognize that by advancing the offset, it's reduced the amount of available
space (bestFit.length). So, the "excessLength" computation below will be
incorrect. If some subsequent element is an exact match for the incorrect
excessLength value, it will overwrite the unused space and clobber whatever
follows. In this case, the thing that got clobbered was the first byte of
EXIF tag 0x9010.
The probability of this happening is small, but non zero. It is just luck that
Sicheng Yang's data sample triggered the issue.
long offset = bestFit.offset;
if ((offset & 1L) != 0) {
offset += 1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (bestFit.length > outputItemLength) {
// not a perfect fit.
final long excessOffset = bestFit.offset + outputItemLength;
final int excessLength = bestFit.length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
I re-wrote the code as follows. It works. Writing a JUnit test for this is
going to be extremely difficult unless we want to include the large test file
in our code distribution (which I think would be a bad idea).
long offset = bestFit.offset;
int length = bestFit.length;
if ((offset & 1L) != 0) {
// offsets have to be at a multiple of 2
offset += 1;
length -=1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (length > outputItemLength) {
// not a perfect fit.
final long excessOffset = offset + outputItemLength;
final int excessLength = length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
was (Author: gwlucas):
Okay, found it.
In the code below, the method looped through all the available free elements
and found one it calls "bestFit". It is going to store the new data into the
available space. But TIFF files have a rule that the offsets have to be an
even multiple of 2. So there's a check to see if the offset is odd and, if it
is, the code advances the offset forward one. The problem is that it doesn't
recognize that by advancing the offset, it's reduced the amount of available
space (bestFit.length). So, the "excessLength" computation below will be
incorrect. If some subsequent element is an exact match for the incorrect
excessLength value, it will overwrite the unused space and clobber whatever
follows. In this case, the thing that got clobbered was the first byte of
EXIF tag 0x9010.
The probability of this happening is small, but non zero. It is just luck that
Sicheng Yang's data sample triggered the issue.
{quote} long offset = bestFit.offset;
if ((offset & 1L) != 0) {
offset += 1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (bestFit.length > outputItemLength) {
// not a perfect fit.
final long excessOffset = bestFit.offset + outputItemLength;
final int excessLength = bestFit.length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
[jira] [Commented] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491005#comment-17491005
]
Gary Lucas commented on IMAGING-319:
Okay, found it.
In the code below, the method looped through all the available free elements
and found one it calls "bestFit". It is going to store the new data into the
available space. But TIFF files have a rule that the offsets have to be an
even multiple of 2. So there's a check to see if the offset is odd and, if it
is, the code advances the offset forward one. The problem is that it doesn't
recognize that by advancing the offset, it's reduced the amount of available
space (bestFit.length). So, the "excessLength" computation below will be
incorrect. If some subsequent element is an exact match for the incorrect
excessLength value, it will overwrite the unused space and clobber whatever
follows. In this case, the thing that got clobbered was the first byte of
EXIF tag 0x9010.
The probability of this happening is small, but non zero. It is just luck that
Sicheng Yang's data sample triggered the issue.
{quote} long offset = bestFit.offset;
if ((offset & 1L) != 0) {
offset += 1;
}
outputItem.setOffset(offset);
unusedElements.remove(bestFit);
if (bestFit.length > outputItemLength) {
// not a perfect fit.
final long excessOffset = bestFit.offset + outputItemLength;
final int excessLength = bestFit.length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
}
{quote}
I re-wrote the code as follows. It works. Writing a JUnit test for this is
going to be extremely difficult.
{quote} unusedElements.remove(bestFit);
long offset = bestFit.offset;
int length = bestFit.length;
if ((offset & 1L) != 0) {
// offsets have to be at a multiple of 2
offset += 1;
length -=1;
}
outputItem.setOffset(offset);
if (length > outputItemLength) {
// not a perfect fit.
final long excessOffset = offset + outputItemLength;
final int excessLength = length - outputItemLength;
unusedElements.add(new TiffElement.Stub(excessOffset,
excessLength));
// make sure the new element is in the correct order.
unusedElements.sort(ELEMENT_SIZE_COMPARATOR);
Collections.reverse(unusedElements);
}
{quote}
> updateExifMetadataLossless lost the first character of a String
> ---
>
> Key: IMAGING-319
> URL: https://issues.apache.org/jira/browse/IMAGING-319
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: JPEG
>Affects Versions: 1.0-alpha2
>Reporter: Sicheng Yang
>Priority: Major
> Attachments: Screen Shot 2021-11-26 at 4.01.06 PM-1.png, Screen Shot
> 2021-11-26 at 4.01.21 PM-1.png, iPhone12-geotag.JPG
>
>
> I try to use TiffOutputSet to generate a new image. However, if a tag that
> contains String, the program may miss the first character of the String.
>
> import java.io.*;
> import org.apache.commons.imaging.ImageReadException;
> import org.apache.commons.imaging.ImageWriteException;
> import org.apache.commons.imaging.Imaging;
> import org.apache.commons.imaging.common.ImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.JpegImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.exif.ExifRewriter;
> import org.apache.commons.imaging.formats.tiff.TiffImageMetadata;
> import org.apache.commons.imaging.formats.tiff.write.TiffOutputSet;
> public class LibraryTest {
> public static void main(String[] args) throws ImageReadException,
> IOException, ImageWriteException {
> File source = new File("./assets/iPhone12-geotag.JPG");
> File result = new
> File("./assets/results/editted-iPhone12-geotag.JPG");
> final ImageMetadata metadata = Imaging.getMetadata(source);
> final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
> final TiffImageMetadata exif = jpegMetadata.getExif();
> TiffOutputSet outputSet = exif.getOutputSet();
> BufferedOutputStream bufferedOutputStream = new
>
[jira] [Commented] (IMAGING-319) updateExifMetadataLossless lost the first character of a String
[
https://issues.apache.org/jira/browse/IMAGING-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490983#comment-17490983
]
Gary Lucas commented on IMAGING-319:
I haven't figured this out yet, but I've narrowed down the cause to
TiffImageWriterLossless. There is a method that attempts to update the file
positions (offsets) of the various EXIF tags. It's called
updateOffsetsSteps().
The code is very confusing. As far as I can tell, at some point some of the
space in the output file is determined to be "unused" and updateOffsetSteps()
attempts to reuse it by finding available space and setting the tag output
position to the available space. If you bypass this operation by adding a
diagnostic call to unusedElements.clear() right after the unusedElements list
is established, everything works fine.
The call stack is basically
ExifRewriter.updateExifMetadataLossles
ExifRewriter.writeExifSegment
TiffImageWriterLossless.write
TiffImageWriterLossless.updateOffsetsStep
> updateExifMetadataLossless lost the first character of a String
> ---
>
> Key: IMAGING-319
> URL: https://issues.apache.org/jira/browse/IMAGING-319
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: JPEG
>Affects Versions: 1.0-alpha2
>Reporter: Sicheng Yang
>Priority: Major
> Attachments: Screen Shot 2021-11-26 at 4.01.06 PM-1.png, Screen Shot
> 2021-11-26 at 4.01.21 PM-1.png, iPhone12-geotag.JPG
>
>
> I try to use TiffOutputSet to generate a new image. However, if a tag that
> contains String, the program may miss the first character of the String.
>
> import java.io.*;
> import org.apache.commons.imaging.ImageReadException;
> import org.apache.commons.imaging.ImageWriteException;
> import org.apache.commons.imaging.Imaging;
> import org.apache.commons.imaging.common.ImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.JpegImageMetadata;
> import org.apache.commons.imaging.formats.jpeg.exif.ExifRewriter;
> import org.apache.commons.imaging.formats.tiff.TiffImageMetadata;
> import org.apache.commons.imaging.formats.tiff.write.TiffOutputSet;
> public class LibraryTest {
> public static void main(String[] args) throws ImageReadException,
> IOException, ImageWriteException {
> File source = new File("./assets/iPhone12-geotag.JPG");
> File result = new
> File("./assets/results/editted-iPhone12-geotag.JPG");
> final ImageMetadata metadata = Imaging.getMetadata(source);
> final JpegImageMetadata jpegMetadata = (JpegImageMetadata) metadata;
> final TiffImageMetadata exif = jpegMetadata.getExif();
> TiffOutputSet outputSet = exif.getOutputSet();
> BufferedOutputStream bufferedOutputStream = new
> BufferedOutputStream(new FileOutputStream(result));
> new ExifRewriter().updateExifMetadataLossless(source,
> bufferedOutputStream, outputSet);
> }
> }
>
> This is the sample code.
> Tag value in original image
> !image-2021-11-26-16-01-58-645.png!
> Tag value in output image
> !image-2021-11-26-16-04-12-185.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (IMAGING-327) Rename setExif method in TiffImagingParameters
[
https://issues.apache.org/jira/browse/IMAGING-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484408#comment-17484408
]
Gary Lucas commented on IMAGING-327:
The new TiffImagingParameters class offers a welcome improvement over the
previous API, but I suggest that we should rename one of the methods to better
reflect what it does.
When creating a new TIFF file, an application passes in a new TiffDirectory
using an instance of TiffOutputSet.
{code:java}
TiffOutputSet tiffOutputSet = new TiffOutputSet();
tiffOutputSet.addDirectory(tiffDirectory);
TiffImagingParameters tiffParams = new TiffImagingParameters();
tiffParams.setExif(tiffOutputSet); {code}
I propose to change the name of the setExif() method to be setOutputSet().
This change will affect some of the unit tests as well as the main API.
Although output sets may include TIFF tags related to the EXIF standard, not
all output sets are EXIF data.
I believe that the setExif() method owes its name to the code used in the
legacy implementation and the API that has now been replaced. SInce we are
improving the API, it makes sense to change it. Here's an example of the
legacy version:
{code:java}
TiffOutputSet tiffOutputSet = new TiffOutputSet();
tiffOutputSet.addDirectory(tiffDirectory);
params.put(ImagingConstants.PARAM_KEY_EXIF, tiffOutputSet); {code}
> Rename setExif method in TiffImagingParameters
> --
>
> Key: IMAGING-327
> URL: https://issues.apache.org/jira/browse/IMAGING-327
> Project: Commons Imaging
> Issue Type: Improvement
> Components: Format: TIFF
>Reporter: Gary Lucas
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (IMAGING-327) Rename setExif method in TiffImagingParameters
Gary Lucas created IMAGING-327: -- Summary: Rename setExif method in TiffImagingParameters Key: IMAGING-327 URL: https://issues.apache.org/jira/browse/IMAGING-327 Project: Commons Imaging Issue Type: Improvement Components: Format: TIFF Reporter: Gary Lucas -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IMAGING-320) Read TIFFs with 32-bit integer samples
Gary Lucas created IMAGING-320: -- Summary: Read TIFFs with 32-bit integer samples Key: IMAGING-320 URL: https://issues.apache.org/jira/browse/IMAGING-320 Project: Commons Imaging Issue Type: New Feature Components: Format: TIFF Reporter: Gary Lucas Issue 266 added the ability to read numerical data from TIFF files that gave 16-bit integer samples. This feature addressed data products such as the Shuttle Radar Topography Mission (SRTM) which provided high-resolution terrestrial elevation data. Recently, I encountered an elevation product that used 32-bit integer samples. I propose to enhance the numerical-data functions to read 32-bit data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437495#comment-17437495 ] Gary Lucas edited comment on IMAGING-316 at 11/2/21, 6:06 PM: -- Short answer, the BigTIFF specification simply calls for changes in the way file-positions are specified in a file. It doesn't change the internal representation of content. A TIFF file can contain multiple bundled images, and their individual formats would not change. However, to provide a bit of background... TIFF actually is an image file specification. And it also supports numeric grids (i.e. Earth surface elevation data sets). TIFF can support data using various data compression methods, of which JPEG is just one. In the case of TIFF, the file does not so much contain a JPEG image as it contains an image and uses JPEG-based data compression to store it. And, again, all images in a TIFF file (whether one or many) are all in a TIFF-specified data format. As an aside, right now there is a Jira item in place to upgrade Commons Imaging to handle TIFF files that contain JPEG-style images, see [Imaging-194|https://issues.apache.org/jira/browse/IMAGING-194] I wish it was as simple as pumping the content through a conventional JPEG API. Although the TIFF format does call for JPEG methods to be used on TIFF files, the internal representation is different enough from the JPEG standard to mess things up. I have only just started looking at that one and have no idea what the level of effort is going to be. But, in answer to your question, I believe that particular issue would be independent of BigTIFF. was (Author: gwlucas): Short answer, the BigTIFF specification simply calls for changes in the way file-positions are specified in a file. It doesn't change the internal representation of content. A TIFF file can contain multiple bundled images, and their individual formats would not change. However, to provide a bit of background... TIFF actually is an image file specification. And it also supports numeric grids (i.e. Earth surface elevation data sets). TIFF can support data using various data compression methods, of which JPEG is just one. In the case of TIFF, the file does not so much contain a JPEG image as it contains an image and uses JPEG-based data compression to store it. And, again, all images in a TIFF file (whether one or many) are all in a TIFF-specified data format. As an aside, right now there is a Jira item in place to upgrade Commons Imaging to handle TIFF files that contain JPEG-style images, see [Imaging-194 |https://issues.apache.org/jira/browse/IMAGING-194 I wish it was as simple as pumping the content through a conventional JPEG API. Although the TIFF format does call for JPEG methods to be used on TIFF files, the internal representation is different enough from the JPEG standard to mess things up. I have only just started looking at that one and have no idea what the level of effort is going to be. But, in answer to your question, I believe that particular issue would be independent of BigTIFF. > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types.
[jira] [Comment Edited] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437495#comment-17437495 ] Gary Lucas edited comment on IMAGING-316 at 11/2/21, 6:04 PM: -- Short answer, the BigTIFF specification simply calls for changes in the way file-positions are specified in a file. It doesn't change the internal representation of content. A TIFF file can contain multiple bundled images, and their individual formats would not change. However, to provide a bit of background... TIFF actually is an image file specification. And it also supports numeric grids (i.e. Earth surface elevation data sets). TIFF can support data using various data compression methods, of which JPEG is just one. In the case of TIFF, the file does not so much contain a JPEG image as it contains an image and uses JPEG-based data compression to store it. And, again, all images in a TIFF file (whether one or many) are all in a TIFF-specified data format. As an aside, right now there is a Jira item in place to upgrade Commons Imaging to handle TIFF files that contain JPEG-style images, see [Imaging-194 |https://issues.apache.org/jira/browse/IMAGING-194 I wish it was as simple as pumping the content through a conventional JPEG API. Although the TIFF format does call for JPEG methods to be used on TIFF files, the internal representation is different enough from the JPEG standard to mess things up. I have only just started looking at that one and have no idea what the level of effort is going to be. But, in answer to your question, I believe that particular issue would be independent of BigTIFF. was (Author: gwlucas): Short answer, the BigTIFF specification simply calls for changes in the way file-positions are specified in a file. It doesn't change the internal representation of content. A TIFF file can contain multiple bundled images, and their individual formats would not change. However, to provide a bit of background... TIFF actually is an image file specification. And it also supports numeric grids (i.e. Earth surface elevation data sets). TIFF can support data using various data compression methods, of which JPEG is just one. In the case of TIFF, the file does not so much contain a JPEG image as it contains an image and uses JPEG-based data compression to store it. And, again, all images in a TIFF file (whether one or many) are all in a TIFF-specified data format. As an aside, right now there is a Jira item in place to upgrade Commons Imaging to handle TIFF files that contain JPEG-style images. I wish it was as simple as pumping the content through a conventional JPEG API. Although the TIFF format does call for JPEG methods to be used on TIFF files, the internal representation is different enough from the JPEG standard to mess things up. I have only just started looking at that one and have no idea what the level of effort is going to be. But, in answer to your question, I believe that particular issue would be independent of BigTIFF. > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types. And it is > possible that a file address with the high bit set might
[jira] [Commented] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437495#comment-17437495 ] Gary Lucas commented on IMAGING-316: Short answer, the BigTIFF specification simply calls for changes in the way file-positions are specified in a file. It doesn't change the internal representation of content. A TIFF file can contain multiple bundled images, and their individual formats would not change. However, to provide a bit of background... TIFF actually is an image file specification. And it also supports numeric grids (i.e. Earth surface elevation data sets). TIFF can support data using various data compression methods, of which JPEG is just one. In the case of TIFF, the file does not so much contain a JPEG image as it contains an image and uses JPEG-based data compression to store it. And, again, all images in a TIFF file (whether one or many) are all in a TIFF-specified data format. As an aside, right now there is a Jira item in place to upgrade Commons Imaging to handle TIFF files that contain JPEG-style images. I wish it was as simple as pumping the content through a conventional JPEG API. Although the TIFF format does call for JPEG methods to be used on TIFF files, the internal representation is different enough from the JPEG standard to mess things up. I have only just started looking at that one and have no idea what the level of effort is going to be. But, in answer to your question, I believe that particular issue would be independent of BigTIFF. > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types. And it is > possible that a file address with the high bit set might be incorrectly > interpreted as a negative number. So I will be taking a look at the code to > make sure all file addresses are properly masked when they are handed over to > Java. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMAGING-316) Support the BigTIFF file format
[ https://issues.apache.org/jira/browse/IMAGING-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-316: --- Description: Traditional TIFF files address file position in bytes using 32-bit integers. This approach automatically limits the maximum size of a TIFF file to 4 GB. The BigTIFF specification (formalized in 2011) uses 64-bit integers to address file positions, and thus supports much larger files. I propose that a future release of Commons Imaging would benefit from supporting BigTIFF. The level of effort for this implementation may be large. In terms of creating JUnit tests to support this effort, note that just because a file uses the BigTIFF specification doesn't mean that the file has to be super large. It should be possible to create BigTIFF test files that are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean that massive files will need to be included in the Commons Imaging distribution. Finally, it is reasonable to ask if anyone would actually need images that were so large that they couldn't fit within 4 GB. The short answer is that some folks in the Geographic Information Systems (GIS) community do work with images (or data sets) that large and, also, that some systems produce images in BigTIFF format even when ordinary TIFF would suffice. P.S. It might be work investigating whether the existing Imaging library actually supports the full 32-bit address space of a conventional TIFF. Regrettably, Java doesn't support unsigned integer types. And it is possible that a file address with the high bit set might be incorrectly interpreted as a negative number. So I will be taking a look at the code to make sure all file addresses are properly masked when they are handed over to Java. was: Traditional TIFF files address file position in bytes using 32-bit integers. This approach automatically limits the maximum size of a TIFF file to 4 GB. The BigTIFF specification (formalized in 2011) uses 64-bit integers to address file positions, and thus supports much larger files. I propose that a future release of Commons Imaging would benefit from supporting BigTIFF. The level of effort for this implementation may be large. In terms of creating JUnit tests to support this effort, note that just because a file uses the BigTIFF specification doesn't mean that the file has to be super large. It should be possible to create BigTIFF test files that are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean that massive files will need to be included in the Commons Imaging distribution. P.S. It might be work investigating whether the existing Imaging library actually supports the full 32-bit address space of a conventional TIFF. Regrettably, Java doesn't support unsigned integer types. And it is possible that a file address with the high bit set might be incorrectly interpreted as a negative number. So I will be taking a look at the code to make sure all file addresses are properly masked when they are handed over to Java. > Support the BigTIFF file format > --- > > Key: IMAGING-316 > URL: https://issues.apache.org/jira/browse/IMAGING-316 > Project: Commons Imaging > Issue Type: New Feature >Affects Versions: 1.x >Reporter: Gary Lucas >Priority: Major > > Traditional TIFF files address file position in bytes using 32-bit integers. > This approach automatically limits the maximum size of a TIFF file to 4 GB. > The BigTIFF specification (formalized in 2011) uses 64-bit integers to > address file positions, and thus supports much larger files. I propose that > a future release of Commons Imaging would benefit from supporting BigTIFF. > The level of effort for this implementation may be large. > In terms of creating JUnit tests to support this effort, note that just > because a file uses the BigTIFF specification doesn't mean that the file has > to be super large. It should be possible to create BigTIFF test files that > are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean > that massive files will need to be included in the Commons Imaging > distribution. > Finally, it is reasonable to ask if anyone would actually need images that > were so large that they couldn't fit within 4 GB. The short answer is that > some folks in the Geographic Information Systems (GIS) community do work with > images (or data sets) that large and, also, that some systems produce images > in BigTIFF format even when ordinary TIFF would suffice. > > P.S. It might be work investigating whether the existing Imaging library > actually supports the full 32-bit address space of a conventional TIFF. > Regrettably, Java doesn't support unsigned integer types. And it is > possible that a
[jira] [Created] (IMAGING-316) Support the BigTIFF file format
Gary Lucas created IMAGING-316: -- Summary: Support the BigTIFF file format Key: IMAGING-316 URL: https://issues.apache.org/jira/browse/IMAGING-316 Project: Commons Imaging Issue Type: New Feature Affects Versions: 1.x Reporter: Gary Lucas Traditional TIFF files address file position in bytes using 32-bit integers. This approach automatically limits the maximum size of a TIFF file to 4 GB. The BigTIFF specification (formalized in 2011) uses 64-bit integers to address file positions, and thus supports much larger files. I propose that a future release of Commons Imaging would benefit from supporting BigTIFF. The level of effort for this implementation may be large. In terms of creating JUnit tests to support this effort, note that just because a file uses the BigTIFF specification doesn't mean that the file has to be super large. It should be possible to create BigTIFF test files that are only a few kilobytes. Thus supporting BigTIFF does not necessarily mean that massive files will need to be included in the Commons Imaging distribution. P.S. It might be work investigating whether the existing Imaging library actually supports the full 32-bit address space of a conventional TIFF. Regrettably, Java doesn't support unsigned integer types. And it is possible that a file address with the high bit set might be incorrectly interpreted as a negative number. So I will be taking a look at the code to make sure all file addresses are properly masked when they are handed over to Java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-313) Provide summary of GeoTIFF tags in example TIFF-dump application
[
https://issues.apache.org/jira/browse/IMAGING-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429778#comment-17429778
]
Gary Lucas edited comment on IMAGING-313 at 10/17/21, 11:36 PM:
The existing Apache Commons Imaging distribution includes an example
application that opens a TIFF image file and extracts the metadata ("tags" in
the TIFF parlance) for inspection. I propose to extend the utility to include
a summary of GeoTIFF-specific tags.
GeoTIFFs are an important class of TIFF files that are used to show imagery
that has a geographic basis. They include Satellite images, aerial photographs,
digitized maps, and even some numerical data such as elevations. The GeoTIFF
standard is a well-known, though slightly complex specification.
The current example application, ReadTagsAndImages.java, prints all tags in the
TIFF file, but the GeoTIFF related information is presented as abstract
numerical values. You can interpret these values if you have a copy of the
GeoTIFF documentation handy, but it is a tedious process. I propose to add
logic to the tag-reading application to format some of that GeoTIFF information
as human-friendly strings.
Here's an example of a numeric data file giving high-resolution elevation data
(some tags omitted). The GeoKeyDirectoryTag is essentially a dictionary giving
a guide to the content to follow. In this case, it consists of 36 integer
value. In the Summary of GeoTIFF Elements, the proposed application
interprets those integer constants as named strings.
{code:java}
Directory 0 Numeric raster data, description: Root
256 (0x100: ImageWidth): 10812 (1 Short)
257 (0x101: ImageLength): 10812 (1 Short)
[snip]
34735 (0x87af: GeoKeyDirectoryTag): 1, 1, 0, 8, 1024, 0, 1, 2, 1025, 0, 1, 1,
(36 elements)
34737 (0x87b1: GeoAsciiParamsTag): 'NAD83|' (7 ASCII)
42112 (0xa480: GDALMetadata): '
NONE
NONE Provide summary of GeoTIFF tags in example TIFF-dump application
>
>
> Key: IMAGING-313
> URL: https://issues.apache.org/jira/browse/IMAGING-313
> Project: Commons Imaging
> Issue Type: New Feature
> Components: Format: TIFF
>Affects Versions: 1.0-alpha3
>Reporter: Gary Lucas
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-313) Provide summary of GeoTIFF tags in example TIFF-dump application
[
https://issues.apache.org/jira/browse/IMAGING-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429778#comment-17429778
]
Gary Lucas edited comment on IMAGING-313 at 10/17/21, 11:35 PM:
The existing Apache Commons Imaging distribution includes an example
application that opens a TIFF image file and extracts the metadata ("tags" in
the TIFF parlance) for inspection. I propose to extend the utility to include
a summary of GeoTIFF-specific tags.
GeoTIFFs are an important class of TIFF files that are used to show imagery
that has a geographic basis. They include Satellite images, aerial photographs,
digitized maps, and even some numerical data such as elevations. The GeoTIFF
standard is a well-known, though slightly complex specification.
The current example application, ReadTagsAndImages.java, prints all tags in the
TIFF file, but the GeoTIFF related information is presented as abstract
numerical values. I propose to add logic to the tag-reading application to
format some of that GeoTIFF information as human-friendly strings.
Here's an example of a numeric data file giving high-resolution elevation data
(some tags omitted). The GeoKeyDirectoryTag is essentially a dictionary giving
a guide to the content to follow. In this case, it consists of 36 integer
value. In the Summary of GeoTIFF Elements, the proposed application
interprets those integer constants as named strings.
{code:java}
Directory 0 Numeric raster data, description: Root
256 (0x100: ImageWidth): 10812 (1 Short)
257 (0x101: ImageLength): 10812 (1 Short)
[snip]
34735 (0x87af: GeoKeyDirectoryTag): 1, 1, 0, 8, 1024, 0, 1, 2, 1025, 0, 1, 1,
(36 elements)
34737 (0x87b1: GeoAsciiParamsTag): 'NAD83|' (7 ASCII)
42112 (0xa480: GDALMetadata): '
NONE
NONE Provide summary of GeoTIFF tags in example TIFF-dump application
>
>
> Key: IMAGING-313
> URL: https://issues.apache.org/jira/browse/IMAGING-313
> Project: Commons Imaging
> Issue Type: New Feature
> Components: Format: TIFF
>Affects Versions: 1.0-alpha3
>Reporter: Gary Lucas
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-313) Provide summary of GeoTIFF tags in example TIFF-dump application
[
https://issues.apache.org/jira/browse/IMAGING-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429778#comment-17429778
]
Gary Lucas edited comment on IMAGING-313 at 10/17/21, 11:33 PM:
The existing Apache Commons Imaging distribution includes an example
application that opens a TIFF image file and extracts the metadata ("tags" in
the TIFF parlance) for inspection. I propose to extend the utility to include
a summary of GeoTIFF-specific tags.
GeoTIFFs are an important class of TIFF files that are used to show imagery
that has a geographic basis. They include Satellite images, aerial photographs,
digitized maps, and even some numerical data such as elevations. The GeoTIFF
standard is well-known, though slightly complex specification.
The current example application, ReadTagsAndImages.java, prints all tags in the
TIFF file, but the GeoTIFF related information is presented as abstract
numerical values. I propose to add logic to the tag-reading application to
format some of that GeoTIFF information as human-friendly strings.
Here's an example of a numeric data file giving high-resolution elevation data
(some tags omitted). The GeoKeyDirectoryTag is essentially a dictionary giving
a guide to the content to follow. In this case, it consists of 36 integer
value. In the Summary of GeoTIFF Elements, the proposed application
interprets those integer constants as named strings.
{code:java}
Directory 0 Numeric raster data, description: Root
256 (0x100: ImageWidth): 10812 (1 Short)
257 (0x101: ImageLength): 10812 (1 Short)
[snip]
34735 (0x87af: GeoKeyDirectoryTag): 1, 1, 0, 8, 1024, 0, 1, 2, 1025, 0, 1, 1,
(36 elements)
34737 (0x87b1: GeoAsciiParamsTag): 'NAD83|' (7 ASCII)
42112 (0xa480: GDALMetadata): '
NONE
NONE Provide summary of GeoTIFF tags in example TIFF-dump application
>
>
> Key: IMAGING-313
> URL: https://issues.apache.org/jira/browse/IMAGING-313
> Project: Commons Imaging
> Issue Type: New Feature
> Components: Format: TIFF
>Affects Versions: 1.0-alpha3
>Reporter: Gary Lucas
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-313) Provide summary of GeoTIFF tags in example TIFF-dump application
[
https://issues.apache.org/jira/browse/IMAGING-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429778#comment-17429778
]
Gary Lucas edited comment on IMAGING-313 at 10/17/21, 11:32 PM:
The existing Apache Commons Imaging distribution includes an example
application that opens a TIFF image file and extracts the metadata ("tags" in
the TIFF parlance) for inspection. I propose to extend the utility to include
a summary of GeoTIFF-specific tags.
GeoTIFFs are an important class of TIFF files that are used to show imagery
that has a geographic basis. They include Satellite images, aerial photographs,
digitized maps, and even some numerical data such as elevations. The GeoTIFF
standard is well-known, though slightly complex specification.
The current example application, ReadTagsAndImages.java, prints all tags in the
TIFF file, but the GeoTIFF related information is presented as abstract
numerical values. I propose to add logic to the tag-reading application to
format some of that GeoTIFF information as human-friendly strings.
Here's an example of a numeric data file giving high-resolution elevation data
(some tags omitted). The GeoKeyDirectoryTag is essentially a dictionary giving
a guide to the content to follow. In this case, it consists of 36 integer
value. In the Summary of GeoTIFF Elements, the proposed application
interprets those integer constants as named strings.
{code:java}
Directory 0 Numeric raster data, description: Root
256 (0x100: ImageWidth): 10812 (1 Short)
257 (0x101: ImageLength): 10812 (1 Short)
[snip]
34735 (0x87af: GeoKeyDirectoryTag): 1, 1, 0, 8, 1024, 0, 1, 2, 1025, 0, 1, 1,
(36 elements)
34737 (0x87b1: GeoAsciiParamsTag): 'NAD83|' (7 ASCII)
42112 (0xa480: GDALMetadata): '
NONE
NONE Provide summary of GeoTIFF tags in example TIFF-dump application
>
>
> Key: IMAGING-313
> URL: https://issues.apache.org/jira/browse/IMAGING-313
> Project: Commons Imaging
> Issue Type: New Feature
> Components: Format: TIFF
>Affects Versions: 1.0-alpha3
>Reporter: Gary Lucas
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IMAGING-313) Provide summary of GeoTIFF tags in example TIFF-dump application
[
https://issues.apache.org/jira/browse/IMAGING-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429778#comment-17429778
]
Gary Lucas commented on IMAGING-313:
The existing Apache Commons Imaging distribution includes an example
application that opens a TIFF image file and extracts the metadata ("tags" in
the TIFF parlance) for inspection. I propose to extend the utility to include
a summary of GeoTIFF-specific tags.
GeoTIFFs are an important class of TIFF files that are used to show imagery
that has a geographic basis. They include Satellite images, aerial photographs,
digitized maps, and even some numerical data such as elevations. The GeoTIFF
standard is well-known, though slightly complex specification.
The current example application, ReadTagsAndImages.java, prints all tags in the
TIFF file, but the GeoTIFF related information is presented as abstract
numerical values. I propose to add logic to the tag-reading application to
format some of that GeoTIFF information as human-friendly strings.
Here's an example of a numeric data file giving high-resolution elevation data
(some tags omitted). The GeoKeyDirectoryTag is essentially a dictionary giving
a guide to the content to follow. In this case, it consists of 36 integer
value. In the Summary of GeoTIFF Elements, the proposed application
interprets those integer constants as named strings.
{code:java}
Directory 0 Numeric raster data, description: Root
256 (0x100: ImageWidth): 10812 (1 Short)
257 (0x101: ImageLength): 10812 (1 Short)
[snip]
34735 (0x87af: GeoKeyDirectoryTag): 1, 1, 0, 8, 1024, 0, 1, 2, 1025, 0, 1, 1,
(36 elements)
34737 (0x87b1: GeoAsciiParamsTag): 'NAD83|' (7 ASCII)
42112 (0xa480: GDALMetadata): '
NONE Provide summary of GeoTIFF tags in example TIFF-dump application
>
>
> Key: IMAGING-313
> URL: https://issues.apache.org/jira/browse/IMAGING-313
> Project: Commons Imaging
> Issue Type: New Feature
> Components: Format: TIFF
>Affects Versions: 1.0-alpha3
>Reporter: Gary Lucas
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IMAGING-313) Provide summary of GeoTIFF tags in example TIFF-dump application
Gary Lucas created IMAGING-313: -- Summary: Provide summary of GeoTIFF tags in example TIFF-dump application Key: IMAGING-313 URL: https://issues.apache.org/jira/browse/IMAGING-313 Project: Commons Imaging Issue Type: New Feature Components: Format: TIFF Affects Versions: 1.0-alpha3 Reporter: Gary Lucas -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-194) Tiff with JPEG,Zip compression fails to decompress
[
https://issues.apache.org/jira/browse/IMAGING-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428230#comment-17428230
]
Gary Lucas edited comment on IMAGING-194 at 10/13/21, 2:27 PM:
---
Does anyone have some manageable-sized, public domain files that we can use to
develop/test this capability?
Does anyone have a current need for this feature?
Through my job, I've got some interest in accessing JPEG based TIFFs. The U.S.
National Weather Service has started posting some satellite images as GeoTIFFs
containing JPEG-formatted data. I will be investigating the level-of-effort for
implementing the ability to read such files. But the task may be out-of-scope
for what I am willing to take on unless there are a significant number of
people interested in the feature.
You can find the satellite images of interest at
[GOES-East - Latest CONUS Images - NOAA / NESDIS /
STAR|https://www.star.nesdis.noaa.gov/GOES/conus.php?sat=G16]
and
[GOES Imagery Viewer - NOAA / NESDIS /
STAR|https://www.star.nesdis.noaa.gov/GOES/]
Some of the satellite-derived imagery is really quite beautiful.
was (Author: gwlucas):
Does anyone have some manageable-sized, public domain files that we can use to
develop/test this capability?
> Tiff with JPEG,Zip compression fails to decompress
> --
>
> Key: IMAGING-194
> URL: https://issues.apache.org/jira/browse/IMAGING-194
> Project: Commons Imaging
> Issue Type: Improvement
> Components: Format: TIFF
>Affects Versions: 1.0-alpha1
>Reporter: Satya Deep Maheshwari
>Priority: Major
>
> Tiff with JPEG, Zip compression fails to decompress with the below exception:
> {code}
> org.apache.commons.imaging.ImageReadException: Tiff: unknown/unsupported
> compression: 7
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReader.decompress(DataReader.java:215)
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReaderStrips.readImageData(DataReaderStrips.java:210)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:650)
> at
> org.apache.commons.imaging.formats.tiff.TiffDirectory.getTiffImage(TiffDirectory.java:157)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:463)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1407)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1370)
> {code}
> From the
> [documentation|https://commons.apache.org/proper/commons-imaging/formatsupport.html]
> , it seems this compression format is not supported. Excerpt from the
> document below:
> {quote}
> Supported through version 6.0. TIFFs is a open-ended container format, so
> it's not possible to support every possibly variation. Supports Bi-Level,
> Palette/Indexed, RGB, CMYK, YCbCr, CIELab and LOGLUV images. Supports reading
> and writing LZW, CCITT Modified Huffman/Group 3/Group 4, and Packbits/RLE
> compression. Notably missing other forms of compression though, including
> JPEG. Supports reading Tiled images.
> {quote}
> This ticket is logged to add JPEG/Zip compression format support.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IMAGING-194) Tiff with JPEG,Zip compression fails to decompress
[
https://issues.apache.org/jira/browse/IMAGING-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428230#comment-17428230
]
Gary Lucas commented on IMAGING-194:
Does anyone have some manageable-sized, public domain files that we can use to
develop/test this capability?
> Tiff with JPEG,Zip compression fails to decompress
> --
>
> Key: IMAGING-194
> URL: https://issues.apache.org/jira/browse/IMAGING-194
> Project: Commons Imaging
> Issue Type: Improvement
> Components: Format: TIFF
>Affects Versions: 1.0-alpha1
>Reporter: Satya Deep Maheshwari
>Priority: Major
>
> Tiff with JPEG, Zip compression fails to decompress with the below exception:
> {code}
> org.apache.commons.imaging.ImageReadException: Tiff: unknown/unsupported
> compression: 7
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReader.decompress(DataReader.java:215)
> at
> org.apache.commons.imaging.formats.tiff.datareaders.DataReaderStrips.readImageData(DataReaderStrips.java:210)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:650)
> at
> org.apache.commons.imaging.formats.tiff.TiffDirectory.getTiffImage(TiffDirectory.java:157)
> at
> org.apache.commons.imaging.formats.tiff.TiffImageParser.getBufferedImage(TiffImageParser.java:463)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1407)
> at
> org.apache.commons.imaging.Imaging.getBufferedImage(Imaging.java:1370)
> {code}
> From the
> [documentation|https://commons.apache.org/proper/commons-imaging/formatsupport.html]
> , it seems this compression format is not supported. Excerpt from the
> document below:
> {quote}
> Supported through version 6.0. TIFFs is a open-ended container format, so
> it's not possible to support every possibly variation. Supports Bi-Level,
> Palette/Indexed, RGB, CMYK, YCbCr, CIELab and LOGLUV images. Supports reading
> and writing LZW, CCITT Modified Huffman/Group 3/Group 4, and Packbits/RLE
> compression. Notably missing other forms of compression though, including
> JPEG. Supports reading Tiled images.
> {quote}
> This ticket is logged to add JPEG/Zip compression format support.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IMAGING-127) API to get a single image should allow choosing which image
[ https://issues.apache.org/jira/browse/IMAGING-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427769#comment-17427769 ] Gary Lucas commented on IMAGING-127: I'm don't know what the level of interest is for this issue. But I do know of an example application in the "test" hierarchy that shows one way to extract images and metadata. It uses low-level calls and is a bit specialized for general application work. It might help get you started on your own implementation. Directory: commons-imaging-master\src\test\java\org\apache\commons\imaging\examples\tiff File ReadTagsAndImages.java > API to get a single image should allow choosing which image > --- > > Key: IMAGING-127 > URL: https://issues.apache.org/jira/browse/IMAGING-127 > Project: Commons Imaging > Issue Type: Improvement >Reporter: Trejkaz >Priority: Major > Fix For: Patch Needed > > Attachments: 2472527552.gif, Wakarusa2015-0001.mpo, june 1 part I.tif > > > getBufferedImage() only returns the first image. There should be a way to > retrieve any image by index (and by extension, an API to get the image count.) > getBufferedImages() cannot be used for large multi-page TIFF files, because > creating that many BufferedImage objects causes an OutOfMemoryError. > (For that matter, a method to get a scaled down copy would be useful as well, > as some formats can optimise that not to retrieve all the data, but also it > means you can reduce the memory usage for absolutely massive images.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-311) Read TIFFs with multiple floating-point samples
[ https://issues.apache.org/jira/browse/IMAGING-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426357#comment-17426357 ] Gary Lucas commented on IMAGING-311: I've made some progress on this issue and have submitted a pull-request to Commons Imaging. A bit more work is still required (mostly additional JUnit tests and a few missing accessor methods). This change will enable Imaging to support additional data sets such as the National Bathymetry System GeoTIFF raster products. These products give ocean depth as a combination of depth and accuracy. The attached picture illustrates the concept. The terrestrial features are from work that was done for Imaging-251, but the oceanographic information comes from multi-variable GeoTIFFs that were not previously accessible by the Imaging API (or any other Java API that I know of). !NBS_Example.jpg! I built the imagery using a combination of Commons Imaging, some open-source software I wrote to demonstrate [Shaded Relief Rendering Techniques,|https://gwlucastrig.github.io/gridfour/notes/ElevationGeoTiff1.html] and some geographic mapping modules I developed for my employer's Java-based [wXstation|http://www.sonalysts.com/products/wxstation/] product. > Read TIFFs with multiple floating-point samples > --- > > Key: IMAGING-311 > URL: https://issues.apache.org/jira/browse/IMAGING-311 > Project: Commons Imaging > Issue Type: New Feature > Components: Format: TIFF >Affects Versions: 1.0-alpha3 > Environment: > [IMAGING-251|https://issues.apache.org/jira/browse/IMAGING-251] >Reporter: Gary Lucas >Priority: Major > Attachments: NBS_Example.jpg > > > I propose to extend Commons Imaging to support reading TIFF files that > contain floating-point formats that feature more than one sample. The > ability to support floating-point samples was introduced in > [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But that > implementation was limited to only those files that provided a single sample > per raster cell (e.g. "a single sample per pixel"). The ability to read files > with multiple samples per raster cell would extend the usefulness of the > Commons Imaging API, particularly for geophysical applications. > If anyone knows of good sources for test TIFF files that use this format, > please let me know. > *Background* > In addition to conventional image data, TIFF files can provide floating-point > numerical information. This feature is often used for geophysical data (in > GeoTIFF files), but can also be applied to other uses. Although the existing > implementation can support files which give a single value per raster cell, > there are some products that carry multiple samples per cell. Examples > include products that give both a measured value and a corresponding accuracy > estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical > products give vectors (gravitational potential, wind vectors, ocean currents, > etc.). > Changes would involve extensions to the classes in the TIFF datareader > package as well as the TiffRasterData class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMAGING-311) Read TIFFs with multiple floating-point samples
[ https://issues.apache.org/jira/browse/IMAGING-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-311: --- Attachment: NBS_Example.jpg > Read TIFFs with multiple floating-point samples > --- > > Key: IMAGING-311 > URL: https://issues.apache.org/jira/browse/IMAGING-311 > Project: Commons Imaging > Issue Type: New Feature > Components: Format: TIFF >Affects Versions: 1.0-alpha3 > Environment: > [IMAGING-251|https://issues.apache.org/jira/browse/IMAGING-251] >Reporter: Gary Lucas >Priority: Major > Attachments: NBS_Example.jpg > > > I propose to extend Commons Imaging to support reading TIFF files that > contain floating-point formats that feature more than one sample. The > ability to support floating-point samples was introduced in > [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But that > implementation was limited to only those files that provided a single sample > per raster cell (e.g. "a single sample per pixel"). The ability to read files > with multiple samples per raster cell would extend the usefulness of the > Commons Imaging API, particularly for geophysical applications. > If anyone knows of good sources for test TIFF files that use this format, > please let me know. > *Background* > In addition to conventional image data, TIFF files can provide floating-point > numerical information. This feature is often used for geophysical data (in > GeoTIFF files), but can also be applied to other uses. Although the existing > implementation can support files which give a single value per raster cell, > there are some products that carry multiple samples per cell. Examples > include products that give both a measured value and a corresponding accuracy > estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical > products give vectors (gravitational potential, wind vectors, ocean currents, > etc.). > Changes would involve extensions to the classes in the TIFF datareader > package as well as the TiffRasterData class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-312) Alpha-channel setting not interpreted from ExtraSamples tag
[ https://issues.apache.org/jira/browse/IMAGING-312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413677#comment-17413677 ] Gary Lucas commented on IMAGING-312: Here's an example comparing the image cited above as rendered by Windows Photo Viewer and the new version of the Commons Imaging API that I will be submitting in a couple of days. You can see the effect of misinterpreting the 4th byte for each pixel as giving alpha values. In reality, the metadata in the TIFF file indicates that the 4th byte should be ignored. !Imaging312.png! > Alpha-channel setting not interpreted from ExtraSamples tag > --- > > Key: IMAGING-312 > URL: https://issues.apache.org/jira/browse/IMAGING-312 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0-alpha2 > Environment: >Reporter: Gary Lucas >Priority: Major > Attachments: Imaging312.png > > > Commons Imaging sometimes misinterprets TIFF files that have 4-byte RGB > samples but do not define alpha. In some cases, these images are treated as > semi-transparent when they should be opaque. Commons Imaging is not unique > in this regard... Windows Photo Viewer does the same thing. > The TIFF specification allows RGB images to be encoded with 4-bytes per > pixel. It would be natural to assume (as Commons Imaging does) that the 4th > byte is the alpha channel and that it would have values of 0xff in the case > where pixels were opaque. However, the interpretation of the 4th byte depends > on information in the TIFF "ExtraSamples" tag. > It turns out that there are images in-the-wild that use 4 bytes, but populate > the 4th byte with junk values. For example, there are a number of older > aerial photographs from the US Geological Survey (USGS) that do this. These > images give an ExtraSamples tag with a value of zero. But the TIFF > specification calls for images to be treated as having alpha channels only if > the ExtraSamples field carries a value of either 1 or 2. When ExtraSamples > has a value of 0, the 4th byte is to be ignored. > So while the USGS TIFF files are in compliance with the TIFF specification, > they use an unintuitive behavior. Because the Commons Imaging library > assumes that the 4th byte would be specified with valid-alpha values, it does > not render the images correctly. > There are many examples of this phenomenon on the USGS Earth Explorer > website. One specific example: > * High Resolution Orthoimagery > * Dataset: 201203_connecticut_state_lot1_ct_0x3000m_utm_cnir > * Entity: 2818289_18TYL425825 > * File: 18tyl425825.tif > *Proposed Fix* > I propose to do the following: > * Extend the TiffImageParser logic for detecting alpha to assume hasAlpha is > true if and only if the ExtraSamples tag is supplied and contains values 1 or > 2. > * Provide a hasAlpha accessor for the ImageBuilder class (is should really > have one anyway) > * Enhance the DataReaderStrips and DataReaderTiles classes to check hasAlpha > when processing RGB images that have 4 samples per pixel samples. > *Concerns* > At this time, I am not sure what to do if an RGB TIFF image uses 4-samples > per pixel but the ExtraSamples tag is not provided. At this time, I have not > seen an example of this, but my collection of sample TIFF files is rather > narrow and I would not rule it out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMAGING-312) Alpha-channel setting not interpreted from ExtraSamples tag
[ https://issues.apache.org/jira/browse/IMAGING-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-312: --- Attachment: Imaging312.png > Alpha-channel setting not interpreted from ExtraSamples tag > --- > > Key: IMAGING-312 > URL: https://issues.apache.org/jira/browse/IMAGING-312 > Project: Commons Imaging > Issue Type: Bug > Components: Format: TIFF >Affects Versions: 1.0-alpha2 > Environment: >Reporter: Gary Lucas >Priority: Major > Attachments: Imaging312.png > > > Commons Imaging sometimes misinterprets TIFF files that have 4-byte RGB > samples but do not define alpha. In some cases, these images are treated as > semi-transparent when they should be opaque. Commons Imaging is not unique > in this regard... Windows Photo Viewer does the same thing. > The TIFF specification allows RGB images to be encoded with 4-bytes per > pixel. It would be natural to assume (as Commons Imaging does) that the 4th > byte is the alpha channel and that it would have values of 0xff in the case > where pixels were opaque. However, the interpretation of the 4th byte depends > on information in the TIFF "ExtraSamples" tag. > It turns out that there are images in-the-wild that use 4 bytes, but populate > the 4th byte with junk values. For example, there are a number of older > aerial photographs from the US Geological Survey (USGS) that do this. These > images give an ExtraSamples tag with a value of zero. But the TIFF > specification calls for images to be treated as having alpha channels only if > the ExtraSamples field carries a value of either 1 or 2. When ExtraSamples > has a value of 0, the 4th byte is to be ignored. > So while the USGS TIFF files are in compliance with the TIFF specification, > they use an unintuitive behavior. Because the Commons Imaging library > assumes that the 4th byte would be specified with valid-alpha values, it does > not render the images correctly. > There are many examples of this phenomenon on the USGS Earth Explorer > website. One specific example: > * High Resolution Orthoimagery > * Dataset: 201203_connecticut_state_lot1_ct_0x3000m_utm_cnir > * Entity: 2818289_18TYL425825 > * File: 18tyl425825.tif > *Proposed Fix* > I propose to do the following: > * Extend the TiffImageParser logic for detecting alpha to assume hasAlpha is > true if and only if the ExtraSamples tag is supplied and contains values 1 or > 2. > * Provide a hasAlpha accessor for the ImageBuilder class (is should really > have one anyway) > * Enhance the DataReaderStrips and DataReaderTiles classes to check hasAlpha > when processing RGB images that have 4 samples per pixel samples. > *Concerns* > At this time, I am not sure what to do if an RGB TIFF image uses 4-samples > per pixel but the ExtraSamples tag is not provided. At this time, I have not > seen an example of this, but my collection of sample TIFF files is rather > narrow and I would not rule it out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-266) Read integer data from GeoTIFFS
[
https://issues.apache.org/jira/browse/IMAGING-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413674#comment-17413674
]
Gary Lucas commented on IMAGING-266:
Well, it took a couple of tries, but I have just submitted changes to address
this issue. This new feature allows developers to use the Commons Imaging API
to access the extensive set of high-resolution, global elevation data available
through the Shuttle Radar Topography Mission (SRTM) as well as many other
geophysical data sources.
Here's an example image produced from a TIFF file that contains elevation data
for a 1-degree square in the vicinity of Auckland, New Zealand. The original
TIFF file gives a 3601-by-3601 grid of elevation data points at a 1 second of
arc spacing (about 30 meters). This image shown below is reduced to 25 percent
of the original size.
!Imaging266_SRTM.jpg!
The image above was produced using a modified version of my DemoCOG
application. I've written a web article describing the techniques used for
rendering numerical data from GeoTIFFs. If you are interested, you can read it
at [https://gwlucastrig.github.io/gridfour/notes/ElevationGeoTiff1.html]
An earlier version of the Imaging API implemented a method called
getFloatingPointRasterData() to read data from TIFF files that contained
floating-point data. However, the elevation data in SRTM products is stored in
a short-integer format. So the access method has been changed to handle both
floating-point and integral data types and its name has been changed to
getRasterData().
The example code below shows how to use the API:
{code:java}
import java.io.File;
import org.apache.commons.imaging.FormatCompliance;
import org.apache.commons.imaging.common.bytesource.ByteSourceFile;
import org.apache.commons.imaging.formats.tiff.TiffContents;
import org.apache.commons.imaging.formats.tiff.TiffDirectory;
import org.apache.commons.imaging.formats.tiff.TiffRasterData;
import org.apache.commons.imaging.formats.tiff.TiffRasterDataType;
import org.apache.commons.imaging.formats.tiff.TiffReader;
public class ExampleRasterReader {
public static void main(String[] args) throws Exception {
File target = new File(args[0]);
ByteSourceFile byteSource = new ByteSourceFile(target);
// Establish a TiffReader. This is just a simple constructor that
// does not actually access the file. So the application cannot
// obtain the byteOrder, or other details, until the contents has
// been read.
TiffReader tiffReader = new TiffReader(true);
// Read the directories in the TIFF file. Directories are the
// main data element of a TIFF file. They usually include an image
// element, but sometimes just carry metadata. This example
// reads all the directories in the file. Typically, reading
// the directories is not a time-consuming operation.
TiffContents contents = tiffReader.readDirectories(
byteSource,
true, // indicates that application should read image data, if
present
FormatCompliance.getDefault());
// Read the first Image File Directory (IFD) in the file. A practical
// implementation could use any of the directories in the file.
// By convention, the main payload (image or raster data) is
// stored in the first TIFF directory with optional thumbnail images
// or metadata directories to follow.
TiffDirectory directory = contents.directories.get(0);
// Check that the first directory in the file has raster data.
if (!directory.hasTiffRasterData()) {
System.out.println(
"Specified directory does not contain raster data");
System.exit(-1);
}
// read all the raster data for the first directory.
// The return value may carry short integers or single-precision
// floating point values. The optional parameter, which is set to
// null in this example, could be used to fetch only a subset
// of the overall data.
TiffRasterData raster = directory.getRasterData(null);
// THe data type may be integral or floating point.
TiffRasterDataType rasterDataType = raster.getDataType();
System.out.println("Data type for raster: " + rasterDataType.name());
int width = raster.getWidth();
int height = raster.getHeight();
int nSamples = 0;
double sumSamples = 0;
if (rasterDataType == TiffRasterDataType.INTEGER) {
// data type is integral, so we use getIntValue()
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
int sample = raster.getIntValue(x, y);
if (sample > 0) {
nSamples++;
sumSamples +=
[jira] [Updated] (IMAGING-266) Read integer data from GeoTIFFS
[ https://issues.apache.org/jira/browse/IMAGING-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-266: --- Attachment: Imaging266_SRTM.jpg > Read integer data from GeoTIFFS > > > Key: IMAGING-266 > URL: https://issues.apache.org/jira/browse/IMAGING-266 > Project: Commons Imaging > Issue Type: New Feature > Components: Format: TIFF >Affects Versions: 1.0-alpha3 >Reporter: Gary Lucas >Assignee: Bruno P. Kinoshita >Priority: Major > Fix For: 1.0-alpha3 > > Attachments: Imaging266_SRTM.jpg > > Time Spent: 5.5h > Remaining Estimate: 0h > > I recently discovered that there is a large amount of digital elevation data > available in the form of 16-bit integer coded data in GeoTIFF files (TIFF > files with geographic tags). I propose to enhance the Commons Imaging API to > read these files. This work will be similar to the work I did for reading > floating-point raster data under ISSUE-251. > Available data include the nearly-global coverage of one-second of arc > elevation data produced from the Shuttle Radar Topography Mission (SRTM) and > other sources. These products give grids of elevation data with a 30 meter > cell spacing for most of the world's land masses. They are available at NASA > Earthdata and Japan Space Systems websites, see > [https://asterweb.jpl.nasa.gov/gdem.asp|https://asterweb.jpl.nasa.gov/gdem.asp] > There is also a ocean bathymetry data set available in this format at > [http://www.shadedrelief.com/blue-earth/] > This new feature will continue to expand the usefulness of the Commons > Imaging API in accessing GeoTIFF products. > Request for Feedback > So far, the data products I've found (ASTER and Blue Earth Bathymetry) give > elevation and ocean depth data in meters recorded as a short integer. I > haven't found an example of where the 32-bit integer format is used. For > now, I am planning on only coding the 16-bit integer variation. Does anyone > know if the 32-bit version is worth supporting? My criteria for determining > that would be based on whether there is a significant number of projects > using that format (life is too short to chase rarely used data formats). > Currently, one of the code-analysis operations conducted by the Commons > Imaging build process is coverage by JUnit tests. Lacking any test data for > the 32-bit case, I am reluctant to include it in the code base because it > would mean putting uncovered code into the distribution. > Also, I am wondering about the best design for the access API. The current > TiffImageParser class has a method called getFloatingPointRasterData() that > returns an instance of TiffRasterData. TiffRasterData is pretty much > hard-wired to floating point data. I am thinking of creating a new method > called getIntegerRasterData() that would return an instance of a new class > called TiffIntegerRasterData. Does this seem reasonable? I considered trying > to combine both kinds of results into a unified class and method, but it > seems more unwieldy than useful. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMAGING-312) Alpha-channel setting not interpreted from ExtraSamples tag
[ https://issues.apache.org/jira/browse/IMAGING-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-312: --- Description: Commons Imaging sometimes misinterprets TIFF files that have 4-byte RGB samples but do not define alpha. In some cases, these images are treated as semi-transparent when they should be opaque. Commons Imaging is not unique in this regard... Windows Photo Viewer does the same thing. The TIFF specification allows RGB images to be encoded with 4-bytes per pixel. It would be natural to assume (as Commons Imaging does) that the 4th byte is the alpha channel and that it would have values of 0xff in the case where pixels were opaque. However, the interpretation of the 4th byte depends on information in the TIFF "ExtraSamples" tag. It turns out that there are images in-the-wild that use 4 bytes, but populate the 4th byte with junk values. For example, there are a number of older aerial photographs from the US Geological Survey (USGS) that do this. These images give an ExtraSamples tag with a value of zero. But the TIFF specification calls for images to be treated as having alpha channels only if the ExtraSamples field carries a value of either 1 or 2. When ExtraSamples has a value of 0, the 4th byte is to be ignored. So while the USGS TIFF files are in compliance with the TIFF specification, they use an unintuitive behavior. Because the Commons Imaging library assumes that the 4th byte would be specified with valid-alpha values, it does not render the images correctly. There are many examples of this phenomenon on the USGS Earth Explorer website. One specific example: * High Resolution Orthoimagery * Dataset: 201203_connecticut_state_lot1_ct_0x3000m_utm_cnir * Entity: 2818289_18TYL425825 * File: 18tyl425825.tif *Proposed Fix* I propose to do the following: * Extend the TiffImageParser logic for detecting alpha to assume hasAlpha is true if and only if the ExtraSamples tag is supplied and contains values 1 or 2. * Provide a hasAlpha accessor for the ImageBuilder class (is should really have one anyway) * Enhance the DataReaderStrips and DataReaderTiles classes to check hasAlpha when processing RGB images that have 4 samples per pixel samples. *Concerns* At this time, I am not sure what to do if an RGB TIFF image uses 4-samples per pixel but the ExtraSamples tag is not provided. At this time, I have not seen an example of this, but my collection of sample TIFF files is rather narrow and I would not rule it out. was: Commons Imaging sometimes misinterprets TIFF files that have 4-byte RGB samples but do not define alpha. In some cases, these images are treated as semi-transparent when they should be opaque. Commons Imaging is not unique in this regard... Windows Photo Viewer does the same thing. The TIFF specification allows RGB images to be encoded with 4-bytes per pixel. It would be natural to assume (as Commons Imaging does) that the 4th byte is the alpha channel and that it would have values of 0xff in the case where pixels were opaque. However, the interpretation of the 4th byte depends on information in the TIFF "ExtraSamples" tag. It turns out that there are images in-the-wild that use 4 bytes, but populate the 4th byte with junk values. For example, there are a number of older aerial photographs from the US Geological Survey (USGS) that do this. These images give an ExtraSamples tag with a value of zero. But the TIFF specification calls for images to be treated as having alpha channels only if the ExtraSamples field carries a value of either 1 or 2. When ExtraSamples has a value of 0, the 4th byte is to be ignored. There are many examples of this phenomenon on the USGS Earth Explorer website. One specific example: * High Resolution Orthoimagery * Dataset: 201203_connecticut_state_lot1_ct_0x3000m_utm_cnir * Entity: 2818289_18TYL425825 * File: 18tyl425825.tif *Proposed Fix* I propose to do the following: * Extend the TiffImageParser logic for detecting alpha to assume hasAlpha is true if and only if the ExtraSamples tag is supplied and contains values 1 or 2. * Provide a hasAlpha accessor for the ImageBuilder class (is should really have one anyway) * Enhance the DataReaderStrips and DataReaderTiles classes to check hasAlpha when processing RGB images that have 4 samples per pixel samples. *Concerns* At this time, I am not sure what to do if an RGB TIFF image uses 4-samples per pixel but the ExtraSamples tag is not provided. At this time, I have not seen an example of this, but my collection of sample TIFF files is rather narrow and I would not rule it out. > Alpha-channel setting not interpreted from ExtraSamples tag > --- > > Key: IMAGING-312 > URL: https://issues.apache.org/jira/browse/IMAGING-312 >
[jira] [Created] (IMAGING-312) Alpha-channel setting not interpreted from ExtraSamples tag
Gary Lucas created IMAGING-312: -- Summary: Alpha-channel setting not interpreted from ExtraSamples tag Key: IMAGING-312 URL: https://issues.apache.org/jira/browse/IMAGING-312 Project: Commons Imaging Issue Type: Bug Components: Format: TIFF Affects Versions: 1.0-alpha2 Environment: Reporter: Gary Lucas Commons Imaging sometimes misinterprets TIFF files that have 4-byte RGB samples but do not define alpha. In some cases, these images are treated as semi-transparent when they should be opaque. Commons Imaging is not unique in this regard... Windows Photo Viewer does the same thing. The TIFF specification allows RGB images to be encoded with 4-bytes per pixel. It would be natural to assume (as Commons Imaging does) that the 4th byte is the alpha channel and that it would have values of 0xff in the case where pixels were opaque. However, the interpretation of the 4th byte depends on information in the TIFF "ExtraSamples" tag. It turns out that there are images in-the-wild that use 4 bytes, but populate the 4th byte with junk values. For example, there are a number of older aerial photographs from the US Geological Survey (USGS) that do this. These images give an ExtraSamples tag with a value of zero. But the TIFF specification calls for images to be treated as having alpha channels only if the ExtraSamples field carries a value of either 1 or 2. When ExtraSamples has a value of 0, the 4th byte is to be ignored. There are many examples of this phenomenon on the USGS Earth Explorer website. One specific example: * High Resolution Orthoimagery * Dataset: 201203_connecticut_state_lot1_ct_0x3000m_utm_cnir * Entity: 2818289_18TYL425825 * File: 18tyl425825.tif *Proposed Fix* I propose to do the following: * Extend the TiffImageParser logic for detecting alpha to assume hasAlpha is true if and only if the ExtraSamples tag is supplied and contains values 1 or 2. * Provide a hasAlpha accessor for the ImageBuilder class (is should really have one anyway) * Enhance the DataReaderStrips and DataReaderTiles classes to check hasAlpha when processing RGB images that have 4 samples per pixel samples. *Concerns* At this time, I am not sure what to do if an RGB TIFF image uses 4-samples per pixel but the ExtraSamples tag is not provided. At this time, I have not seen an example of this, but my collection of sample TIFF files is rather narrow and I would not rule it out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMAGING-311) Read TIFFs with multiple floating-point samples
[ https://issues.apache.org/jira/browse/IMAGING-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-311: --- Description: I propose to extend Commons Imaging to support reading TIFF files that contain floating-point formats that feature more than one sample. The ability to support floating-point samples was introduced in [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But that implementation was limited to only those files that provided a single sample per raster cell (e.g. "a single sample per pixel"). The ability to read files with multiple samples per raster cell would extend the usefulness of the Commons Imaging API, particularly for geophysical applications. If anyone knows of good sources for test TIFF files that use this format, please let me know. *Background* In addition to conventional image data, TIFF files can provide floating-point numerical information. This feature is often used for geophysical data (in GeoTIFF files), but can also be applied to other uses. Although the existing implementation can support files which give a single value per raster cell, there are some products that carry multiple samples per cell. Examples include products that give both a measured value and a corresponding accuracy estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical products give vectors (gravitational potential, wind vectors, ocean currents, etc.). Changes would involve extensions to the classes in the TIFF datareader package as well as the TiffRasterData class. was: I propose to extend Commons Imaging to support reading TIFF files that contain floating-point formats that feature more than one sample. The ability to support floating-point samples was introduced in [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But this implementation was limited to only those files that provided a single sample per raster cell (e.g. "a single sample per pixel"). The ability to read files with multiple samples per raster cell would extend the usefulness of the Commons Imaging API, particularly for geophysical applications. If anyone knows of good sources for test TIFF files that use this format, please let me know. *Background* In addition to conventional image data, TIFF files can provide floating-point numerical information. This feature is often used for geophysical data (in GeoTIFF files), but can also be applied to other uses. Although the existing implementation can support files which give a single value per raster cell, there are some products that carry multiple samples per cell. Examples include products that give both a measured value and a corresponding accuracy estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical products give vectors (gravitational potential, wind vectors, ocean currents, etc.). Changes would involve extensions to the classes in the TIFF datareader package as well as the TiffRasterData class. > Read TIFFs with multiple floating-point samples > --- > > Key: IMAGING-311 > URL: https://issues.apache.org/jira/browse/IMAGING-311 > Project: Commons Imaging > Issue Type: New Feature > Components: Format: TIFF >Affects Versions: 1.0-alpha3 > Environment: > [IMAGING-251|https://issues.apache.org/jira/browse/IMAGING-251] >Reporter: Gary Lucas >Priority: Major > > I propose to extend Commons Imaging to support reading TIFF files that > contain floating-point formats that feature more than one sample. The > ability to support floating-point samples was introduced in > [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But that > implementation was limited to only those files that provided a single sample > per raster cell (e.g. "a single sample per pixel"). The ability to read files > with multiple samples per raster cell would extend the usefulness of the > Commons Imaging API, particularly for geophysical applications. > If anyone knows of good sources for test TIFF files that use this format, > please let me know. > *Background* > In addition to conventional image data, TIFF files can provide floating-point > numerical information. This feature is often used for geophysical data (in > GeoTIFF files), but can also be applied to other uses. Although the existing > implementation can support files which give a single value per raster cell, > there are some products that carry multiple samples per cell. Examples > include products that give both a measured value and a corresponding accuracy > estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical > products give vectors (gravitational potential, wind vectors, ocean currents, > etc.). > Changes would involve extensions to the classes in the TIFF datareader
[jira] [Updated] (IMAGING-311) Read TIFFs with multiple floating-point samples
[ https://issues.apache.org/jira/browse/IMAGING-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-311: --- Description: I propose to extend Commons Imaging to support reading TIFF files that contain floating-point formats that feature more than one sample. The ability to support floating-point samples was introduced in [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But this implementation was limited to only those files that provided a single sample per raster cell (e.g. "a single sample per pixel"). The ability to read files with multiple samples per raster cell would extend the usefulness of the Commons Imaging API, particularly for geophysical applications. If anyone knows of good sources for test TIFF files that use this format, please let me know. *Background* In addition to conventional image data, TIFF files can provide floating-point numerical information. This feature is often used for geophysical data (in GeoTIFF files), but can also be applied to other uses. Although the existing implementation can support files which give a single value per raster cell, there are some products that carry multiple samples per cell. Examples include products that give both a measured value and a corresponding accuracy estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical products give vectors (gravitational potential, wind vectors, ocean currents, etc.). Changes would involve extensions to the classes in the TIFF datareader package as well as the TiffRasterData class. was: I propose to extend Commons Imaging to support reading TIFF files that contain floating-point formats that feature more than one sample. The ability to support floating-point samples was introduced in [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But this implementation was limited to only those files that provided a single sample per raster cell (a single sample per pixel). If anyone knows of good sources for test TIFF files that use this format, please let me know. *Background* In addition to conventional image data, TIFF files can provide floating-point numerical information. This feature is often used for geophysical data (in GeoTIFF files), but can also be applied to other uses. Although the existing implementation can support files which give a single value per raster cell, there are some products that carry multiple samples per cell. Examples include products that give both a measured value and a corresponding accuracy estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical products give vectors (gravitational potential, wind vectors, ocean currents, etc.). Changes would involve extensions to the classes in the TIFF datareader package as well as the TiffRasterData class. > Read TIFFs with multiple floating-point samples > --- > > Key: IMAGING-311 > URL: https://issues.apache.org/jira/browse/IMAGING-311 > Project: Commons Imaging > Issue Type: New Feature > Components: Format: TIFF >Affects Versions: 1.0-alpha3 > Environment: > [IMAGING-251|https://issues.apache.org/jira/browse/IMAGING-251] >Reporter: Gary Lucas >Priority: Major > > I propose to extend Commons Imaging to support reading TIFF files that > contain floating-point formats that feature more than one sample. The > ability to support floating-point samples was introduced in > [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But this > implementation was limited to only those files that provided a single sample > per raster cell (e.g. "a single sample per pixel"). The ability to read files > with multiple samples per raster cell would extend the usefulness of the > Commons Imaging API, particularly for geophysical applications. > If anyone knows of good sources for test TIFF files that use this format, > please let me know. > *Background* > In addition to conventional image data, TIFF files can provide floating-point > numerical information. This feature is often used for geophysical data (in > GeoTIFF files), but can also be applied to other uses. Although the existing > implementation can support files which give a single value per raster cell, > there are some products that carry multiple samples per cell. Examples > include products that give both a measured value and a corresponding accuracy > estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical > products give vectors (gravitational potential, wind vectors, ocean currents, > etc.). > Changes would involve extensions to the classes in the TIFF datareader > package as well as the TiffRasterData class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMAGING-311) Read TIFFs with multiple floating-point samples
[ https://issues.apache.org/jira/browse/IMAGING-311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-311: --- Description: I propose to extend Commons Imaging to support reading TIFF files that contain floating-point formats that feature more than one sample. The ability to support floating-point samples was introduced in [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But this implementation was limited to only those files that provided a single sample per raster cell (a single sample per pixel). If anyone knows of good sources for test TIFF files that use this format, please let me know. *Background* In addition to conventional image data, TIFF files can provide floating-point numerical information. This feature is often used for geophysical data (in GeoTIFF files), but can also be applied to other uses. Although the existing implementation can support files which give a single value per raster cell, there are some products that carry multiple samples per cell. Examples include products that give both a measured value and a corresponding accuracy estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical products give vectors (gravitational potential, wind vectors, ocean currents, etc.). Changes would involve extensions to the classes in the TIFF datareader package as well as the TiffRasterData class. was:I propose to extend Commons Imaging to support reading TIFF files that contain floating-point formats that feature more than one sample. The ability to support floating-point samples was introduced in > Read TIFFs with multiple floating-point samples > --- > > Key: IMAGING-311 > URL: https://issues.apache.org/jira/browse/IMAGING-311 > Project: Commons Imaging > Issue Type: New Feature > Components: Format: TIFF >Affects Versions: 1.0-alpha3 > Environment: > [IMAGING-251|https://issues.apache.org/jira/browse/IMAGING-251] >Reporter: Gary Lucas >Priority: Major > > I propose to extend Commons Imaging to support reading TIFF files that > contain floating-point formats that feature more than one sample. The > ability to support floating-point samples was introduced in > [ISSUE-251|https://issues.apache.org/jira/browse/IMAGING-251]. But this > implementation was limited to only those files that provided a single sample > per raster cell (a single sample per pixel). > If anyone knows of good sources for test TIFF files that use this format, > please let me know. > *Background* > In addition to conventional image data, TIFF files can provide floating-point > numerical information. This feature is often used for geophysical data (in > GeoTIFF files), but can also be applied to other uses. Although the existing > implementation can support files which give a single value per raster cell, > there are some products that carry multiple samples per cell. Examples > include products that give both a measured value and a corresponding accuracy > estimate (i.e. 245.6 meters plus or minus 0.5 meters). Some geophysical > products give vectors (gravitational potential, wind vectors, ocean currents, > etc.). > Changes would involve extensions to the classes in the TIFF datareader > package as well as the TiffRasterData class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IMAGING-311) Read TIFFs with multiple floating-point samples
Gary Lucas created IMAGING-311: -- Summary: Read TIFFs with multiple floating-point samples Key: IMAGING-311 URL: https://issues.apache.org/jira/browse/IMAGING-311 Project: Commons Imaging Issue Type: New Feature Components: Format: TIFF Affects Versions: 1.0-alpha3 Environment: [IMAGING-251|https://issues.apache.org/jira/browse/IMAGING-251] Reporter: Gary Lucas I propose to extend Commons Imaging to support reading TIFF files that contain floating-point formats that feature more than one sample. The ability to support floating-point samples was introduced in -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-285) Decoding of Rational Numbers broken when large values present
[ https://issues.apache.org/jira/browse/IMAGING-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397715#comment-17397715 ] Gary Lucas commented on IMAGING-285: I've got some fixes to the code base to correct the GPS Info behaviors reported by this issue. They involve changes to the RationalNumber class, but also involve some changes that have bubbled upward into some of the modules that call them including ByteConversions and various Tag-related classes. I should be posting the new logic as soon as I put together some appropriate JUnit tests. > Decoding of Rational Numbers broken when large values present > - > > Key: IMAGING-285 > URL: https://issues.apache.org/jira/browse/IMAGING-285 > Project: Commons Imaging > Issue Type: Bug > Components: imaging.common.* >Affects Versions: 1.0-alpha2 >Reporter: John Andrade >Priority: Major > Attachments: DJI_0267 cropped.JPG > > Original Estimate: 1h > Remaining Estimate: 1h > > Decoding Lat/Long EXIF data from JPEGs does not work for some values. I have > attached a file where the conversion fails. I used the > getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast methods from the > TiffImageMetaData.GPSInfo class. The values are close, but seemingly off by > a few miles. > I've traced the source and I believe the issue is with how the RationalNumber > class uses two 32-bit signed integers for the numerator and denominator. The > definition of a RATIONAL data type uses 32-bit unsigned numbers. It seems as > if the RationalNumber class already expects this as it has a non-public > static method called factoryMethod to create a RationalNumber from two 64-bit > numbers. > This error is introduced in the ByteConversions class, specifically the > toRational method where it uses the regular RationalNumber constructor and > thus ensures any rational numbers that have numerator or denominator greater > than the max signed 32-bit value will produce invalid values. > I am attaching a JPEG that has this problem. I had to crop it to reduce the > size, but the EXIF data was preserved. > The EXIF GPS data contained in the JPEG is: > GpsLatitudeRef: "N" > GpsLatitude: 38, 1, 36, 1, 4120083230, 7000 > GpsLongitudeRef: "W" > GpsLongitude: 90, 1, 12, 1, 2379156648, 7000 > According to the properties of the image (right-clicking on Windows 10), the > L/L of the image is: > Latitude: 38: 36: 58.85833 > Longitude: 90: 12: 33.98795... (Windows doesn't show E/W) > These values convert to: > 38.616349536627 > -90.2094410978095 > When I use the getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast > methods, I get the following values: > 38.5993060156 > -90.19239757679365 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-285) Decoding of Rational Numbers broken when large values present
[
https://issues.apache.org/jira/browse/IMAGING-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17396933#comment-17396933
]
Gary Lucas commented on IMAGING-285:
I think that the suggestion of using Commons Numbers really belongs in a
separate Jira item as an "enhancement". For now I'm going to limit my changes
to the issue that I understand and save any undertakings of a broader scope for
future work.
As an addendum, it turns out that there are some clear pro's and con's for
Gilles Sadowski's position. I will start with the con.
It turns out that the Commons Numbers API shares the same problem as the
Commons Imaging code. The problem is that the TIFF specification for Rational
elements calls for rational numbers based on two _*unsigned*_ 32-bit integers.
But both Imaging's current RationalNumber class and Number's Fraction class
both assume inputs as signed integers. So computing the latitude using the
parameters from the sample EXIF data in the original post, we find:
{code:java}
int numerator = -174884066; // 0xf5937b1e
int denominator = 7000;
Fraction frac = Fraction.of(numerator, denominator);
System.out.println("frac="+frac.doubleValue());
// The "unsigned" approach-
long n = numerator &0xL;
long d = denominator&0xL;
double lat = (double)n/(double)d;
System.out.println("latitude = "+lat);
Results:
frac = -2.4983438
latitude = 58.85833185714286
{code}
Thus we see that Numbers has its own variation of the problem. Numbers does
have a class called BigInteger which could be used to deal with this. But to me
it seems a little bit over complicated for this requirement.
On the pro side of Mr. Sadowski's suggestion, it looks like the code in Numbers
has received a lot more focused development than the code in Imaging's
RationalNumber class. For example, looking at the code for the Imaging's
RationalNumber class, I spotted a shortcoming:
{code:java}
@Override
public float floatValue() {
return (float) numerator / (float) divisor;
}{code}
The shortcoming here is that an IEEE-754 standard 32-bit floating point value
has only 24 bits of precision in its mantissa while an unsigned integer has 32.
So some of that low-order digits in the numerator and denominator could be
thrown away by the cast even before the division operation is performed. A
better solution would be to do things the way Commons Numbers does, which would
be basically the following:
{code:java}
@Override
public float floatValue() {
return (float) doubleValue();
}
@Override
public double doubleValue() {
return (double) numerator / (double) divisor;
}
{code}
> Decoding of Rational Numbers broken when large values present
> -
>
> Key: IMAGING-285
> URL: https://issues.apache.org/jira/browse/IMAGING-285
> Project: Commons Imaging
> Issue Type: Bug
> Components: imaging.common.*
>Affects Versions: 1.0-alpha2
>Reporter: John Andrade
>Priority: Major
> Attachments: DJI_0267 cropped.JPG
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Decoding Lat/Long EXIF data from JPEGs does not work for some values. I have
> attached a file where the conversion fails. I used the
> getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast methods from the
> TiffImageMetaData.GPSInfo class. The values are close, but seemingly off by
> a few miles.
> I've traced the source and I believe the issue is with how the RationalNumber
> class uses two 32-bit signed integers for the numerator and denominator. The
> definition of a RATIONAL data type uses 32-bit unsigned numbers. It seems as
> if the RationalNumber class already expects this as it has a non-public
> static method called factoryMethod to create a RationalNumber from two 64-bit
> numbers.
> This error is introduced in the ByteConversions class, specifically the
> toRational method where it uses the regular RationalNumber constructor and
> thus ensures any rational numbers that have numerator or denominator greater
> than the max signed 32-bit value will produce invalid values.
> I am attaching a JPEG that has this problem. I had to crop it to reduce the
> size, but the EXIF data was preserved.
> The EXIF GPS data contained in the JPEG is:
> GpsLatitudeRef: "N"
> GpsLatitude: 38, 1, 36, 1, 4120083230, 7000
> GpsLongitudeRef: "W"
> GpsLongitude: 90, 1, 12, 1, 2379156648, 7000
> According to the properties of the image (right-clicking on Windows 10), the
> L/L of the image is:
> Latitude: 38: 36: 58.85833
> Longitude: 90: 12: 33.98795... (Windows doesn't show E/W)
> These values convert to:
> 38.616349536627
> -90.2094410978095
> When I use the getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast
> methods, I get
[jira] [Commented] (IMAGING-285) Decoding of Rational Numbers broken when large values present
[ https://issues.apache.org/jira/browse/IMAGING-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17396661#comment-17396661 ] Gary Lucas commented on IMAGING-285: > Is there some particular feature that would prevent that the latter [Commons >Numbers] be a replacement for the former [existing code]? It's not that at all. I looked at Commons Numbers, and there's a lot to like about it. I really hope you see widespread adoption. In the case of Commons Imaging, I think that the issue is one of project dependencies. I caveat this with saying that I am not in charge of Commons Imaging, so I don't speak for the project. But I am reluctant to add any new dependencies to what should be just a single component API for developers to drop into to their applications. Also, Imaging's RationalNumber class is small enough that there isn't a strong motivation to depend on an external project even though that project would be almost certainly be more carefully written and maintained than the single, specialized class in Commons Imaging. Interestingly, the TIFF standard does not specify the arithmetic to be used for rational numbers. It calls for specifying real numbers using two unsigned 32-bit integers. Let's call them a and b. So the meaning of the computed floating point value (double)a/(double)b is pretty unambiguous. But the Commons Imaging RationalNumber class also has a method that returns the integer value a/b. The TIFF standard doesn't say anything about round-off. But maybe (a+b/2)/b might have been a better solution? I personally think so, but I am not about to mess with the way the code has always worked. At the same time, I have to say that one clear advantage of Commons Numbers is that It would "fill in the blanks" where there were gaps in the specification. Since Numbers specializes in things that Imaging merely uses, I'm sure you guys have worked through the details on operations like that. > Decoding of Rational Numbers broken when large values present > - > > Key: IMAGING-285 > URL: https://issues.apache.org/jira/browse/IMAGING-285 > Project: Commons Imaging > Issue Type: Bug > Components: imaging.common.* >Affects Versions: 1.0-alpha2 >Reporter: John Andrade >Priority: Major > Attachments: DJI_0267 cropped.JPG > > Original Estimate: 1h > Remaining Estimate: 1h > > Decoding Lat/Long EXIF data from JPEGs does not work for some values. I have > attached a file where the conversion fails. I used the > getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast methods from the > TiffImageMetaData.GPSInfo class. The values are close, but seemingly off by > a few miles. > I've traced the source and I believe the issue is with how the RationalNumber > class uses two 32-bit signed integers for the numerator and denominator. The > definition of a RATIONAL data type uses 32-bit unsigned numbers. It seems as > if the RationalNumber class already expects this as it has a non-public > static method called factoryMethod to create a RationalNumber from two 64-bit > numbers. > This error is introduced in the ByteConversions class, specifically the > toRational method where it uses the regular RationalNumber constructor and > thus ensures any rational numbers that have numerator or denominator greater > than the max signed 32-bit value will produce invalid values. > I am attaching a JPEG that has this problem. I had to crop it to reduce the > size, but the EXIF data was preserved. > The EXIF GPS data contained in the JPEG is: > GpsLatitudeRef: "N" > GpsLatitude: 38, 1, 36, 1, 4120083230, 7000 > GpsLongitudeRef: "W" > GpsLongitude: 90, 1, 12, 1, 2379156648, 7000 > According to the properties of the image (right-clicking on Windows 10), the > L/L of the image is: > Latitude: 38: 36: 58.85833 > Longitude: 90: 12: 33.98795... (Windows doesn't show E/W) > These values convert to: > 38.616349536627 > -90.2094410978095 > When I use the getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast > methods, I get the following values: > 38.5993060156 > -90.19239757679365 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-285) Decoding of Rational Numbers broken when large values present
[ https://issues.apache.org/jira/browse/IMAGING-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17396327#comment-17396327 ] Gary Lucas commented on IMAGING-285: No. Thanks for bringing it to my attention. I was vaguely aware that there was some kind of calving process going on in Commons with regard to all things mathematical, but I didn't realize that Commons Numbers existed. The RationalNumbers handling in the TIFF/EXIF formats is pretty narrow and specialized. So it won't benefit from Commons Numbers, but on quick inspection there appears to be a lot of interesting things going on: quaternions, gamma functions. Cool stuff. > Decoding of Rational Numbers broken when large values present > - > > Key: IMAGING-285 > URL: https://issues.apache.org/jira/browse/IMAGING-285 > Project: Commons Imaging > Issue Type: Bug > Components: imaging.common.* >Affects Versions: 1.0-alpha2 >Reporter: John Andrade >Priority: Major > Attachments: DJI_0267 cropped.JPG > > Original Estimate: 1h > Remaining Estimate: 1h > > Decoding Lat/Long EXIF data from JPEGs does not work for some values. I have > attached a file where the conversion fails. I used the > getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast methods from the > TiffImageMetaData.GPSInfo class. The values are close, but seemingly off by > a few miles. > I've traced the source and I believe the issue is with how the RationalNumber > class uses two 32-bit signed integers for the numerator and denominator. The > definition of a RATIONAL data type uses 32-bit unsigned numbers. It seems as > if the RationalNumber class already expects this as it has a non-public > static method called factoryMethod to create a RationalNumber from two 64-bit > numbers. > This error is introduced in the ByteConversions class, specifically the > toRational method where it uses the regular RationalNumber constructor and > thus ensures any rational numbers that have numerator or denominator greater > than the max signed 32-bit value will produce invalid values. > I am attaching a JPEG that has this problem. I had to crop it to reduce the > size, but the EXIF data was preserved. > The EXIF GPS data contained in the JPEG is: > GpsLatitudeRef: "N" > GpsLatitude: 38, 1, 36, 1, 4120083230, 7000 > GpsLongitudeRef: "W" > GpsLongitude: 90, 1, 12, 1, 2379156648, 7000 > According to the properties of the image (right-clicking on Windows 10), the > L/L of the image is: > Latitude: 38: 36: 58.85833 > Longitude: 90: 12: 33.98795... (Windows doesn't show E/W) > These values convert to: > 38.616349536627 > -90.2094410978095 > When I use the getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast > methods, I get the following values: > 38.5993060156 > -90.19239757679365 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-285) Decoding of Rational Numbers broken when large values present
[ https://issues.apache.org/jira/browse/IMAGING-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17396275#comment-17396275 ] Gary Lucas commented on IMAGING-285: Thank you for identifying this and supplying resources that it to be tested. I have a fix in for the RationalNumber class that seems to work. Your diagnosis about it being in the ByteConversions class was correct. I've got a bit more polishing to do on the code before I submit it and I need to go over the case where SRational Number format is used. Quick question: Do you have additional resources for testing, or perhaps a dump of the elements in the TIFF file? This is not a feature that I use, so I would like to get a bit more extensive tests before I call it "done". Every so often, I have to remind myself that TIFF is a very old format and, in fact, dates from before the IEEE-754 standard was universally adopted. It's the only reason I can think of that the Rational Number format would even have come into existence. > Decoding of Rational Numbers broken when large values present > - > > Key: IMAGING-285 > URL: https://issues.apache.org/jira/browse/IMAGING-285 > Project: Commons Imaging > Issue Type: Bug > Components: imaging.common.* >Affects Versions: 1.0-alpha2 >Reporter: John Andrade >Priority: Major > Attachments: DJI_0267 cropped.JPG > > Original Estimate: 1h > Remaining Estimate: 1h > > Decoding Lat/Long EXIF data from JPEGs does not work for some values. I have > attached a file where the conversion fails. I used the > getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast methods from the > TiffImageMetaData.GPSInfo class. The values are close, but seemingly off by > a few miles. > I've traced the source and I believe the issue is with how the RationalNumber > class uses two 32-bit signed integers for the numerator and denominator. The > definition of a RATIONAL data type uses 32-bit unsigned numbers. It seems as > if the RationalNumber class already expects this as it has a non-public > static method called factoryMethod to create a RationalNumber from two 64-bit > numbers. > This error is introduced in the ByteConversions class, specifically the > toRational method where it uses the regular RationalNumber constructor and > thus ensures any rational numbers that have numerator or denominator greater > than the max signed 32-bit value will produce invalid values. > I am attaching a JPEG that has this problem. I had to crop it to reduce the > size, but the EXIF data was preserved. > The EXIF GPS data contained in the JPEG is: > GpsLatitudeRef: "N" > GpsLatitude: 38, 1, 36, 1, 4120083230, 7000 > GpsLongitudeRef: "W" > GpsLongitude: 90, 1, 12, 1, 2379156648, 7000 > According to the properties of the image (right-clicking on Windows 10), the > L/L of the image is: > Latitude: 38: 36: 58.85833 > Longitude: 90: 12: 33.98795... (Windows doesn't show E/W) > These values convert to: > 38.616349536627 > -90.2094410978095 > When I use the getLatitudeAsDegreesNorth and getLongitudeAsDegreesEast > methods, I get the following values: > 38.5993060156 > -90.19239757679365 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-284) Tif file with transparent background result in black background png
[
https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312763#comment-17312763
]
Gary Lucas edited comment on IMAGING-284 at 3/31/21, 11:05 PM:
---
I'm deeply suspicious of this mediafire website. When I attempted to download
your file, they tried to push some adware onto my computer and reconfigured my
Chrome browser search. On the other hand, the file did download, so I took a
look.
I wrote a simple test program that read the TIFF file, obtained a buffered
image, and then drew that image to an "composite" image with a blue background.
Everything seemed to work fine.
{code:java}
public static void main(String[] args) throws IOException, ImageReadException {
File input = new File(args[0]);
Map params = new HashMap<>();
BufferedImage bImage = Imaging.getBufferedImage(input, params);
int w = bImage.getWidth();
int h = bImage.getHeight();
BufferedImage composite = new BufferedImage(w, h,
BufferedImage.TYPE_INT_RGB);
Graphics2D g = composite.createGraphics();
g.setColor(Color.blue);
g.fillRect(0, 0, w + 1, h + 1);
g.drawImage(bImage, 0, 0, w, h, null);
File output = new File("Test.jpg");
System.out.println("Writing image to " + output.getPath());
ImageIO.write(composite, "JPG", output);
}
{code}
!Test.jpg!
was (Author: gwlucas):
I'm deeply suspicious of this mediafire website. When I attempted to download
your file, they tried to push some adware onto my computer and reconfigured my
Chrome browser search. On the other hand, the file did download, so I took a
look.
I wrote a simple test program that read the TIFF file, obtained a buffered
image, and then drew that image to an "composite" image with a blue background.
Everything seemed to work fine.
{code:java}
public static void main(String[] args) throws IOException, ImageReadException {
File input = new File(args[0]);
Map params = new HashMap<>();
BufferedImage bImage = Imaging.getBufferedImage(input, params);
int w = bImage.getWidth();
int h = bImage.getHeight();
BufferedImage composite = new BufferedImage(w, h,
BufferedImage.TYPE_INT_RGB);
Graphics2D g = composite.createGraphics();
g.setColor(Color.blue);
g.fillRect(0, 0, w + 1, h + 1);
g.drawImage(bImage, 0, 0, w, h, null);File output = new
File("Test.jpg");
System.out.println("Writing image to " + output.getPath());
ImageIO.write(composite, "JPG", output);
}
{code}
!Test.jpg!
> Tif file with transparent background result in black background png
> ---
>
> Key: IMAGING-284
> URL: https://issues.apache.org/jira/browse/IMAGING-284
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: PNG
>Reporter: Gaurav Gupta
>Priority: Major
> Attachments: Test.jpg
>
>
> When a tiff file (sample file [0]) with transparent background is converted
> to png file using commons-imaging library, the background in png file is
> black instead of transparent.
> [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IMAGING-284) Tif file with transparent background result in black background png
[
https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312763#comment-17312763
]
Gary Lucas commented on IMAGING-284:
I'm deeply suspicious of this mediafire website. When I attempted to download
your file, they tried to push some adware onto my computer and reconfigured my
Chrome browser search. On the other hand, the file did download, so I took a
look.
I wrote a simple test program that read the TIFF file, obtained a buffered
image, and then drew that image to an "composite" image with a blue background.
Everything seemed to work fine.
{code:java}
public static void main(String[] args) throws IOException, ImageReadException {
File input = new File(args[0]);
Map params = new HashMap<>();
BufferedImage bImage = Imaging.getBufferedImage(input, params);
int w = bImage.getWidth();
int h = bImage.getHeight();
BufferedImage composite = new BufferedImage(w, h,
BufferedImage.TYPE_INT_RGB);
Graphics2D g = composite.createGraphics();
g.setColor(Color.blue);
g.fillRect(0, 0, w + 1, h + 1);
g.drawImage(bImage, 0, 0, w, h, null);File output = new
File("Test.jpg");
System.out.println("Writing image to " + output.getPath());
ImageIO.write(composite, "JPG", output);
}
{code}
!Test.jpg!
> Tif file with transparent background result in black background png
> ---
>
> Key: IMAGING-284
> URL: https://issues.apache.org/jira/browse/IMAGING-284
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: PNG
>Reporter: Gaurav Gupta
>Priority: Major
> Attachments: Test.jpg
>
>
> When a tiff file (sample file [0]) with transparent background is converted
> to png file using commons-imaging library, the background in png file is
> black instead of transparent.
> [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IMAGING-284) Tif file with transparent background result in black background png
[ https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Lucas updated IMAGING-284: --- Attachment: Test.jpg > Tif file with transparent background result in black background png > --- > > Key: IMAGING-284 > URL: https://issues.apache.org/jira/browse/IMAGING-284 > Project: Commons Imaging > Issue Type: Bug > Components: Format: PNG >Reporter: Gaurav Gupta >Priority: Major > Attachments: Test.jpg > > > When a tiff file (sample file [0]) with transparent background is converted > to png file using commons-imaging library, the background in png file is > black instead of transparent. > [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (IMAGING-284) Tif file with transparent background result in black background png
[
https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Lucas updated IMAGING-284:
---
Comment: was deleted
(was: I'm deeply suspicious of this mediafire website. When I attempted to
download your file, they tried to push some adware onto my computer and
reconfigured my Chrome browser search. On the other hand, the file did
download, so I took a look.
I wrote a simple test program that read the TIFF file, obtained a buffered
image, and then drew that image to an "composite" image with a blue background.
Everything seemed to work fine.
The code
{code:java}
public static void main(String[] args) throws IOException, ImageReadException {
File input = new File(args[0]);
Map params = new HashMap<>();
BufferedImage bImage = Imaging.getBufferedImage(input, params);
int w = bImage.getWidth();
int h = bImage.getHeight();
BufferedImage composite = new BufferedImage(w, h,
BufferedImage.TYPE_INT_RGB);
Graphics2D g = composite.createGraphics();
g.setColor(Color.blue);
g.fillRect(0, 0, w + 1, h + 1);
g.drawImage(bImage, 0, 0, w, h, null);File output = new
File("Test.jpg");
System.out.println("Writing image to " + output.getPath());
ImageIO.write(composite, "JPG", output);
}
{code}
)
> Tif file with transparent background result in black background png
> ---
>
> Key: IMAGING-284
> URL: https://issues.apache.org/jira/browse/IMAGING-284
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: PNG
>Reporter: Gaurav Gupta
>Priority: Major
>
> When a tiff file (sample file [0]) with transparent background is converted
> to png file using commons-imaging library, the background in png file is
> black instead of transparent.
> [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-284) Tif file with transparent background result in black background png
[
https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312746#comment-17312746
]
Gary Lucas edited comment on IMAGING-284 at 3/31/21, 11:03 PM:
---
I'm deeply suspicious of this mediafire website. When I attempted to download
your file, they tried to push some adware onto my computer and reconfigured my
Chrome browser search. On the other hand, the file did download, so I took a
look.
I wrote a simple test program that read the TIFF file, obtained a buffered
image, and then drew that image to an "composite" image with a blue background.
Everything seemed to work fine.
The code
{code:java}
public static void main(String[] args) throws IOException, ImageReadException {
File input = new File(args[0]);
Map params = new HashMap<>();
BufferedImage bImage = Imaging.getBufferedImage(input, params);
int w = bImage.getWidth();
int h = bImage.getHeight();
BufferedImage composite = new BufferedImage(w, h,
BufferedImage.TYPE_INT_RGB);
Graphics2D g = composite.createGraphics();
g.setColor(Color.blue);
g.fillRect(0, 0, w + 1, h + 1);
g.drawImage(bImage, 0, 0, w, h, null);File output = new
File("Test.jpg");
System.out.println("Writing image to " + output.getPath());
ImageIO.write(composite, "JPG", output);
}
{code}
was (Author: gwlucas):
I'm deeply suspicious of this mediafire website. When I attempted to download
your file, they tried to push some adware onto my computer and reconfigured my
Chrome browser search. I'm not going back. So in the absence of a better way
of accessing the material, I don't think there's much we can do.
> Tif file with transparent background result in black background png
> ---
>
> Key: IMAGING-284
> URL: https://issues.apache.org/jira/browse/IMAGING-284
> Project: Commons Imaging
> Issue Type: Bug
> Components: Format: PNG
>Reporter: Gaurav Gupta
>Priority: Major
>
> When a tiff file (sample file [0]) with transparent background is converted
> to png file using commons-imaging library, the background in png file is
> black instead of transparent.
> [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-284) Tif file with transparent background result in black background png
[ https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312746#comment-17312746 ] Gary Lucas edited comment on IMAGING-284 at 3/31/21, 10:38 PM: --- I'm deeply suspicious of this mediafire website. When I attempted to download your file, they tried to push some adware onto my computer and reconfigured my Chrome browser search. I'm not going back. So in absence a better way of accessing the material, I don't think there's much we can do. was (Author: gwlucas): I'm deeply suspicious of this mediafire website. When I attempted to download your file, they tried to push some adware onto my computer and reconfigured by Chrome browser search. I'm not going back. So in absence a better way of accessing the material, I don't think there's much we can do. > Tif file with transparent background result in black background png > --- > > Key: IMAGING-284 > URL: https://issues.apache.org/jira/browse/IMAGING-284 > Project: Commons Imaging > Issue Type: Bug > Components: Format: PNG >Reporter: Gaurav Gupta >Priority: Major > > When a tiff file (sample file [0]) with transparent background is converted > to png file using commons-imaging library, the background in png file is > black instead of transparent. > [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-284) Tif file with transparent background result in black background png
[ https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312746#comment-17312746 ] Gary Lucas edited comment on IMAGING-284 at 3/31/21, 10:38 PM: --- I'm deeply suspicious of this mediafire website. When I attempted to download your file, they tried to push some adware onto my computer and reconfigured my Chrome browser search. I'm not going back. So in the absence of a better way of accessing the material, I don't think there's much we can do. was (Author: gwlucas): I'm deeply suspicious of this mediafire website. When I attempted to download your file, they tried to push some adware onto my computer and reconfigured my Chrome browser search. I'm not going back. So in absence a better way of accessing the material, I don't think there's much we can do. > Tif file with transparent background result in black background png > --- > > Key: IMAGING-284 > URL: https://issues.apache.org/jira/browse/IMAGING-284 > Project: Commons Imaging > Issue Type: Bug > Components: Format: PNG >Reporter: Gaurav Gupta >Priority: Major > > When a tiff file (sample file [0]) with transparent background is converted > to png file using commons-imaging library, the background in png file is > black instead of transparent. > [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-284) Tif file with transparent background result in black background png
[ https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312746#comment-17312746 ] Gary Lucas commented on IMAGING-284: I'm deeply suspicious of this mediafire website. When I attempted to download your file, they tried to push some adware onto my computer and reconfigured by Chrome browser search. I'm not going back. So in absence a better way of accessing the material, I don't think there's much we can do. > Tif file with transparent background result in black background png > --- > > Key: IMAGING-284 > URL: https://issues.apache.org/jira/browse/IMAGING-284 > Project: Commons Imaging > Issue Type: Bug > Components: Format: PNG >Reporter: Gaurav Gupta >Priority: Major > > When a tiff file (sample file [0]) with transparent background is converted > to png file using commons-imaging library, the background in png file is > black instead of transparent. > [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IMAGING-284) Tif file with transparent background result in black background png
[ https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310787#comment-17310787 ] Gary Lucas edited comment on IMAGING-284 at 3/29/21, 4:57 PM: -- What version of Commons Imaging were you using? I ask because the code currently in the Github repository should be able to handle TIFF's with transparency. However, that code is not yet been released to the Maven Central Repository. So if you were using the compiled Jars from the Maven Central Repository, they would not handle your image file properly. But if you downloaded code from Github and compiled your own Jars, they ought to work. If they don't then there could be a bug in the implementation. I don't manage the Commons Imaging project, but I did contribute the code enhancements for the TIFF transparency feature. So if there is a problem, I would be anxious to find it. was (Author: gwlucas): What version of Commons Imaging were you using? I ask because the code currently in the Github repository should be able to handle TIFF's with transparency. However, that code is not yet been released to the Maven Central Repository. So if you were using the compiled Jars from the Maven Central Repository, they would not handle your image file properly. But if you downloaded code from Github and compiled your own Jars, they ought to work. If they don't then there could be a bug in the implementation. I don't manage the Commons Imaging project, but I did contribute the code enhancements for the TIFF transparency feature. So if there is a problem, I would be anxious to find it. How big is your test file? If it's not too big, would it be possible to attach it to this Jira issue? I do not have access to mediafire.com > Tif file with transparent background result in black background png > --- > > Key: IMAGING-284 > URL: https://issues.apache.org/jira/browse/IMAGING-284 > Project: Commons Imaging > Issue Type: Bug > Components: Format: PNG >Reporter: Gaurav Gupta >Priority: Major > > When a tiff file (sample file [0]) with transparent background is converted > to png file using commons-imaging library, the background in png file is > black instead of transparent. > [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMAGING-284) Tif file with transparent background result in black background png
[ https://issues.apache.org/jira/browse/IMAGING-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310787#comment-17310787 ] Gary Lucas commented on IMAGING-284: What version of Commons Imaging were you using? I ask because the code currently in the Github repository should be able to handle TIFF's with transparency. However, that code is not yet been released to the Maven Central Repository. So if you were using the compiled Jars from the Maven Central Repository, they would not handle your image file properly. But if you downloaded code from Github and compiled your own Jars, they ought to work. If they don't then there could be a bug in the implementation. I don't manage the Commons Imaging project, but I did contribute the code enhancements for the TIFF transparency feature. So if there is a problem, I would be anxious to find it. How big is your test file? If it's not too big, would it be possible to attach it to this Jira issue? I do not have access to mediafire.com > Tif file with transparent background result in black background png > --- > > Key: IMAGING-284 > URL: https://issues.apache.org/jira/browse/IMAGING-284 > Project: Commons Imaging > Issue Type: Bug > Components: Format: PNG >Reporter: Gaurav Gupta >Priority: Major > > When a tiff file (sample file [0]) with transparent background is converted > to png file using commons-imaging library, the background in png file is > black instead of transparent. > [0] https://www.mediafire.com/file/tl0r9tsvx6a16zg/Test.tif/file -- This message was sent by Atlassian Jira (v8.3.4#803005)
