[GitHub] [parquet-format] emkornfield merged pull request #182: Fix typo under "Unsigned Integers"
emkornfield merged PR #182: URL: https://github.com/apache/parquet-format/pull/182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [parquet-format] emkornfield commented on pull request #182: Fix typo under "Unsigned Integers"
emkornfield commented on PR #182: URL: https://github.com/apache/parquet-format/pull/182#issuecomment-1342078961 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PARQUET-758) [Format] HALF precision FLOAT Logical type
[ https://issues.apache.org/jira/browse/PARQUET-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644544#comment-17644544 ] ASF GitHub Bot commented on PARQUET-758: emkornfield commented on PR #184: URL: https://github.com/apache/parquet-format/pull/184#issuecomment-1341697557 @anjakefala IIUC, I think the prior objection was around not properly floating point sorting for statistics correctly. I think the main thing is to update the specification to say this requires the same sorting logic as float32 and float64. I need to rereview the current state of things, but then I think we can probably try to vote on the mailing list to see if this type is acceptable. I'm not sure on the exact process here (I don't know if implementations are required before a vote). @gszadovszky thoughts? > [Format] HALF precision FLOAT Logical type > -- > > Key: PARQUET-758 > URL: https://issues.apache.org/jira/browse/PARQUET-758 > Project: Parquet > Issue Type: Improvement > Components: parquet-format >Reporter: Julien Le Dem >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [parquet-format] emkornfield commented on pull request #184: PARQUET-758: Add Float16/Half-float logical type
emkornfield commented on PR #184: URL: https://github.com/apache/parquet-format/pull/184#issuecomment-1341697557 @anjakefala IIUC, I think the prior objection was around not properly floating point sorting for statistics correctly. I think the main thing is to update the specification to say this requires the same sorting logic as float32 and float64. I need to rereview the current state of things, but then I think we can probably try to vote on the mailing list to see if this type is acceptable. I'm not sure on the exact process here (I don't know if implementations are required before a vote). @gszadovszky thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PARQUET-758) [Format] HALF precision FLOAT Logical type
[ https://issues.apache.org/jira/browse/PARQUET-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644535#comment-17644535 ] ASF GitHub Bot commented on PARQUET-758: anjakefala commented on PR #184: URL: https://github.com/apache/parquet-format/pull/184#issuecomment-1341655934 Thanks so much for the update @emkornfield! What is the next step I can take? > [Format] HALF precision FLOAT Logical type > -- > > Key: PARQUET-758 > URL: https://issues.apache.org/jira/browse/PARQUET-758 > Project: Parquet > Issue Type: Improvement > Components: parquet-format >Reporter: Julien Le Dem >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [parquet-format] anjakefala commented on pull request #184: PARQUET-758: Add Float16/Half-float logical type
anjakefala commented on PR #184: URL: https://github.com/apache/parquet-format/pull/184#issuecomment-1341655934 Thanks so much for the update @emkornfield! What is the next step I can take? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (PARQUET-2201) Add Stress test for RecordReader SkipRecords
[ https://issues.apache.org/jira/browse/PARQUET-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated PARQUET-2201: Labels: pull-request-available (was: ) > Add Stress test for RecordReader SkipRecords > > > Key: PARQUET-2201 > URL: https://issues.apache.org/jira/browse/PARQUET-2201 > Project: Parquet > Issue Type: Improvement > Components: parquet-cpp >Reporter: fatemah >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Adding a stress test that will call a random sequence of ReadRecords and > SkipRecords. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-758) [Format] HALF precision FLOAT Logical type
[ https://issues.apache.org/jira/browse/PARQUET-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644458#comment-17644458 ] ASF GitHub Bot commented on PARQUET-758: emkornfield commented on PR #184: URL: https://github.com/apache/parquet-format/pull/184#issuecomment-1341351021 Sorry for the delay but PARQUET-1222 has now been merged, so I think this is unblocked. > [Format] HALF precision FLOAT Logical type > -- > > Key: PARQUET-758 > URL: https://issues.apache.org/jira/browse/PARQUET-758 > Project: Parquet > Issue Type: Improvement > Components: parquet-format >Reporter: Julien Le Dem >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-758) [Format] HALF precision FLOAT Logical type
[
https://issues.apache.org/jira/browse/PARQUET-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644457#comment-17644457
]
ASF GitHub Bot commented on PARQUET-758:
emkornfield commented on code in PR #184:
URL: https://github.com/apache/parquet-format/pull/184#discussion_r1042516177
##
src/main/thrift/parquet.thrift:
##
@@ -232,6 +232,7 @@ struct MapType {} // see LogicalTypes.md
struct ListType {}// see LogicalTypes.md
struct EnumType {}// allowed for BINARY, must be encoded with UTF-8
struct DateType {}// allowed for INT32
+struct Float16Type {} // allowed for FIXED[2], must encoded raw FLOAT16 bytes
Review Comment:
yes. BYTE_STREAM_SPLIT
> [Format] HALF precision FLOAT Logical type
> --
>
> Key: PARQUET-758
> URL: https://issues.apache.org/jira/browse/PARQUET-758
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-format
>Reporter: Julien Le Dem
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [parquet-format] emkornfield commented on a diff in pull request #184: PARQUET-758: Add Float16/Half-float logical type
emkornfield commented on code in PR #184:
URL: https://github.com/apache/parquet-format/pull/184#discussion_r1042516177
##
src/main/thrift/parquet.thrift:
##
@@ -232,6 +232,7 @@ struct MapType {} // see LogicalTypes.md
struct ListType {}// see LogicalTypes.md
struct EnumType {}// allowed for BINARY, must be encoded with UTF-8
struct DateType {}// allowed for INT32
+struct Float16Type {} // allowed for FIXED[2], must encoded raw FLOAT16 bytes
Review Comment:
yes. BYTE_STREAM_SPLIT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [parquet-format] emkornfield commented on pull request #184: PARQUET-758: Add Float16/Half-float logical type
emkornfield commented on PR #184: URL: https://github.com/apache/parquet-format/pull/184#issuecomment-1341351021 Sorry for the delay but PARQUET-1222 has now been merged, so I think this is unblocked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PARQUET-1222) Specify a well-defined sorting order for float and double types
[
https://issues.apache.org/jira/browse/PARQUET-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644456#comment-17644456
]
ASF GitHub Bot commented on PARQUET-1222:
-
emkornfield commented on PR #185:
URL: https://github.com/apache/parquet-format/pull/185#issuecomment-1341349709
> I don't feel it would require a formal vote. In my view it was more a
missing part of the spec that is made clear.
It might worth a heads up on the dev list, though.
Replied to my original thread that this has been merged.
> Specify a well-defined sorting order for float and double types
> ---
>
> Key: PARQUET-1222
> URL: https://issues.apache.org/jira/browse/PARQUET-1222
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
>Reporter: Zoltan Ivanfi
>Assignee: Micah Kornfield
>Priority: Critical
> Fix For: format-2.10.0
>
>
> Currently parquet-format specifies the sort order for floating point numbers
> as follows:
> {code:java}
>* FLOAT - signed comparison of the represented value
>* DOUBLE - signed comparison of the represented value
> {code}
> The problem is that the comparison of floating point numbers is only a
> partial ordering with strange behaviour in specific corner cases. For
> example, according to IEEE 754, -0 is neither less nor more than \+0 and
> comparing NaN to anything always returns false. This ordering is not suitable
> for statistics. Additionally, the Java implementation already uses a
> different (total) ordering that handles these cases correctly but differently
> than the C\+\+ implementations, which leads to interoperability problems.
> TypeDefinedOrder for doubles and floats should be deprecated and a new
> TotalFloatingPointOrder should be introduced. The default for writing doubles
> and floats would be the new TotalFloatingPointOrder. This ordering should be
> effective and easy to implement in all programming languages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [parquet-format] emkornfield commented on pull request #185: PARQUET-1222: [Format] Add details about sort order to README.md
emkornfield commented on PR #185: URL: https://github.com/apache/parquet-format/pull/185#issuecomment-1341349709 > I don't feel it would require a formal vote. In my view it was more a missing part of the spec that is made clear. It might worth a heads up on the dev list, though. Replied to my original thread that this has been merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [Format] Clarifying Sort Order Requirements for Floating Points and Logical Types
https://github.com/apache/parquet-format/pull/185 has been merged. On Fri, Nov 4, 2022 at 9:54 PM Micah Kornfield wrote: > A new proposal for adding a logical annotation to support Float16 values > [1] reopened the discussion on specifying how parquet should deal with > edge cases for floating point types (PARQUET-1222 [2]). > > To try to resolve this the consensus from the JIRA is to not try to > specify an ordering when writing but only rules but rather only specify > rules for reading data. The rules where already present in the > parquet.thrift file [3]. They are: > >> >> >>* - If the min is a NaN, it should be ignored. >>* - If the max is a NaN, it should be ignored. >>* - If the min is +0, the row group may contain -0 values as well. >>* - If the max is -0, the row group may contain +0 values as well. >>* - When looking for NaN values, min and max should be ignored. > > > I've created a PR [4] to update README.md in parquet-format that: > 1. Specifies statistics should not be used when a column has an unknown > logical type since correct comparisons cannot be performed. > 2. Specifies the ordering for primitive types and references the > parquet.thrift for the details on how to handle floating point values. > > Feedback and other ideas are welcome. > > Thanks, > Micah > > [1] https://github.com/apache/parquet-format/pull/184 > [2] https://issues.apache.org/jira/browse/PARQUET-1222 > [3] > https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L897 > [4] https://github.com/apache/parquet-format/pull/185 > >
[jira] [Commented] (PARQUET-1222) Specify a well-defined sorting order for float and double types
[
https://issues.apache.org/jira/browse/PARQUET-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1766#comment-1766
]
ASF GitHub Bot commented on PARQUET-1222:
-
gszadovszky commented on PR #185:
URL: https://github.com/apache/parquet-format/pull/185#issuecomment-1341331959
I don't feel it would require a formal vote. In my view it was more a
missing part of the spec that is made clear.
It might worth a heads up on the dev list, though.
> Specify a well-defined sorting order for float and double types
> ---
>
> Key: PARQUET-1222
> URL: https://issues.apache.org/jira/browse/PARQUET-1222
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
>Reporter: Zoltan Ivanfi
>Assignee: Micah Kornfield
>Priority: Critical
> Fix For: format-2.10.0
>
>
> Currently parquet-format specifies the sort order for floating point numbers
> as follows:
> {code:java}
>* FLOAT - signed comparison of the represented value
>* DOUBLE - signed comparison of the represented value
> {code}
> The problem is that the comparison of floating point numbers is only a
> partial ordering with strange behaviour in specific corner cases. For
> example, according to IEEE 754, -0 is neither less nor more than \+0 and
> comparing NaN to anything always returns false. This ordering is not suitable
> for statistics. Additionally, the Java implementation already uses a
> different (total) ordering that handles these cases correctly but differently
> than the C\+\+ implementations, which leads to interoperability problems.
> TypeDefinedOrder for doubles and floats should be deprecated and a new
> TotalFloatingPointOrder should be introduced. The default for writing doubles
> and floats would be the new TotalFloatingPointOrder. This ordering should be
> effective and easy to implement in all programming languages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [parquet-format] gszadovszky commented on pull request #185: PARQUET-1222: [Format] Add details about sort order to README.md
gszadovszky commented on PR #185: URL: https://github.com/apache/parquet-format/pull/185#issuecomment-1341331959 I don't feel it would require a formal vote. In my view it was more a missing part of the spec that is made clear. It might worth a heads up on the dev list, though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PARQUET-1222) Specify a well-defined sorting order for float and double types
[
https://issues.apache.org/jira/browse/PARQUET-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1762#comment-1762
]
ASF GitHub Bot commented on PARQUET-1222:
-
pitrou commented on PR #185:
URL: https://github.com/apache/parquet-format/pull/185#issuecomment-1341327375
> @pitrou thanks for merging, I'm not sure if we needed an official vote on
this?
Oops, that's a good question. I might have merged too quickly. @gszadovszky
What is your take on this?
> Specify a well-defined sorting order for float and double types
> ---
>
> Key: PARQUET-1222
> URL: https://issues.apache.org/jira/browse/PARQUET-1222
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
>Reporter: Zoltan Ivanfi
>Assignee: Micah Kornfield
>Priority: Critical
> Fix For: format-2.10.0
>
>
> Currently parquet-format specifies the sort order for floating point numbers
> as follows:
> {code:java}
>* FLOAT - signed comparison of the represented value
>* DOUBLE - signed comparison of the represented value
> {code}
> The problem is that the comparison of floating point numbers is only a
> partial ordering with strange behaviour in specific corner cases. For
> example, according to IEEE 754, -0 is neither less nor more than \+0 and
> comparing NaN to anything always returns false. This ordering is not suitable
> for statistics. Additionally, the Java implementation already uses a
> different (total) ordering that handles these cases correctly but differently
> than the C\+\+ implementations, which leads to interoperability problems.
> TypeDefinedOrder for doubles and floats should be deprecated and a new
> TotalFloatingPointOrder should be introduced. The default for writing doubles
> and floats would be the new TotalFloatingPointOrder. This ordering should be
> effective and easy to implement in all programming languages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [parquet-format] pitrou commented on pull request #185: PARQUET-1222: [Format] Add details about sort order to README.md
pitrou commented on PR #185: URL: https://github.com/apache/parquet-format/pull/185#issuecomment-1341327375 > @pitrou thanks for merging, I'm not sure if we needed an official vote on this? Oops, that's a good question. I might have merged too quickly. @gszadovszky What is your take on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (PARQUET-1222) Specify a well-defined sorting order for float and double types
[
https://issues.apache.org/jira/browse/PARQUET-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou updated PARQUET-1222:
Fix Version/s: format-2.10.0
> Specify a well-defined sorting order for float and double types
> ---
>
> Key: PARQUET-1222
> URL: https://issues.apache.org/jira/browse/PARQUET-1222
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
>Reporter: Zoltan Ivanfi
>Assignee: Micah Kornfield
>Priority: Critical
> Fix For: format-2.10.0
>
>
> Currently parquet-format specifies the sort order for floating point numbers
> as follows:
> {code:java}
>* FLOAT - signed comparison of the represented value
>* DOUBLE - signed comparison of the represented value
> {code}
> The problem is that the comparison of floating point numbers is only a
> partial ordering with strange behaviour in specific corner cases. For
> example, according to IEEE 754, -0 is neither less nor more than \+0 and
> comparing NaN to anything always returns false. This ordering is not suitable
> for statistics. Additionally, the Java implementation already uses a
> different (total) ordering that handles these cases correctly but differently
> than the C\+\+ implementations, which leads to interoperability problems.
> TypeDefinedOrder for doubles and floats should be deprecated and a new
> TotalFloatingPointOrder should be introduced. The default for writing doubles
> and floats would be the new TotalFloatingPointOrder. This ordering should be
> effective and easy to implement in all programming languages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (PARQUET-1222) Specify a well-defined sorting order for float and double types
[
https://issues.apache.org/jira/browse/PARQUET-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou resolved PARQUET-1222.
-
Resolution: Fixed
> Specify a well-defined sorting order for float and double types
> ---
>
> Key: PARQUET-1222
> URL: https://issues.apache.org/jira/browse/PARQUET-1222
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
>Reporter: Zoltan Ivanfi
>Assignee: Micah Kornfield
>Priority: Critical
>
> Currently parquet-format specifies the sort order for floating point numbers
> as follows:
> {code:java}
>* FLOAT - signed comparison of the represented value
>* DOUBLE - signed comparison of the represented value
> {code}
> The problem is that the comparison of floating point numbers is only a
> partial ordering with strange behaviour in specific corner cases. For
> example, according to IEEE 754, -0 is neither less nor more than \+0 and
> comparing NaN to anything always returns false. This ordering is not suitable
> for statistics. Additionally, the Java implementation already uses a
> different (total) ordering that handles these cases correctly but differently
> than the C\+\+ implementations, which leads to interoperability problems.
> TypeDefinedOrder for doubles and floats should be deprecated and a new
> TotalFloatingPointOrder should be introduced. The default for writing doubles
> and floats would be the new TotalFloatingPointOrder. This ordering should be
> effective and easy to implement in all programming languages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (PARQUET-1222) Specify a well-defined sorting order for float and double types
[
https://issues.apache.org/jira/browse/PARQUET-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou reassigned PARQUET-1222:
---
Assignee: Micah Kornfield
> Specify a well-defined sorting order for float and double types
> ---
>
> Key: PARQUET-1222
> URL: https://issues.apache.org/jira/browse/PARQUET-1222
> Project: Parquet
> Issue Type: Bug
> Components: parquet-format
>Reporter: Zoltan Ivanfi
>Assignee: Micah Kornfield
>Priority: Critical
>
> Currently parquet-format specifies the sort order for floating point numbers
> as follows:
> {code:java}
>* FLOAT - signed comparison of the represented value
>* DOUBLE - signed comparison of the represented value
> {code}
> The problem is that the comparison of floating point numbers is only a
> partial ordering with strange behaviour in specific corner cases. For
> example, according to IEEE 754, -0 is neither less nor more than \+0 and
> comparing NaN to anything always returns false. This ordering is not suitable
> for statistics. Additionally, the Java implementation already uses a
> different (total) ordering that handles these cases correctly but differently
> than the C\+\+ implementations, which leads to interoperability problems.
> TypeDefinedOrder for doubles and floats should be deprecated and a new
> TotalFloatingPointOrder should be introduced. The default for writing doubles
> and floats would be the new TotalFloatingPointOrder. This ordering should be
> effective and easy to implement in all programming languages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [parquet-format] pitrou merged pull request #186: MINOR: Update parquet.thrift to fix a typo
pitrou merged PR #186: URL: https://github.com/apache/parquet-format/pull/186 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [parquet-format] pitrou merged pull request #176: MINOR: Typo in parquet.thrift
pitrou merged PR #176: URL: https://github.com/apache/parquet-format/pull/176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [parquet-format] gszadovszky commented on pull request #186: MINOR: Update parquet.thrift to fix a typo
gszadovszky commented on PR #186: URL: https://github.com/apache/parquet-format/pull/186#issuecomment-1340556422 @pitrou, Jira is required to track potential discussions etc. I don't think we need one for a typo fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [parquet-format] pitrou commented on pull request #186: MINOR: Update parquet.thrift to fix a typo
pitrou commented on PR #186: URL: https://github.com/apache/parquet-format/pull/186#issuecomment-1340552504 @gszadovszky Is creating a JIRA required for such a trivial fix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
