[ https://issues.apache.org/jira/browse/PARQUET-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689729#comment-17689729 ]
ASF GitHub Bot commented on PARQUET-2244: ----------------------------------------- zhongyujiang commented on PR #1028: URL: https://github.com/apache/parquet-mr/pull/1028#issuecomment-1433005114 > I don't know if there is a downstream that relies on Parquet judge value <> null as TRUE instead of UNKNOW, I guess that might be in some non-ansi standard engines. I just want to make sure it's safe to revert this, I did some search and find this: https://stackoverflow.com/questions/129077/null-values-inside-not-in-clause Seems SQL Server 2005 has a switch `ansi_null` to control wheather the result of `value <> null` is `TRUE` or `UNKNOWN`, when `ansi_nulls` is off, `3 <> null` is true. Though I didn't find such a switch in engines like Hive or Spark. Do you have any comments on this? Or do you think it's worth caring about? @gszadovszky @wgtmac @huaxingao. > Dictionary filter may skip row-groups incorrectly when evaluating notIn > ----------------------------------------------------------------------- > > Key: PARQUET-2244 > URL: https://issues.apache.org/jira/browse/PARQUET-2244 > Project: Parquet > Issue Type: Bug > Components: parquet-mr > Affects Versions: 1.12.2 > Reporter: Yujiang Zhong > Assignee: Yujiang Zhong > Priority: Major > > Dictionary filter may skip row-groups incorrectly when evaluating `notIn` on > optional columns with null values. Here is an example: > Say there is a optional column `c1` with all pages dict encoded, `c1` has and > only has two distinct values: ['foo', null], and the predicate is `c1 not > in ('foo', 'bar')`. > Now dictionary filter may skip this row-group that is actually should not be > skipped, because there are nulls in the column. > > This is a bug similar to #1510. -- This message was sent by Atlassian Jira (v8.20.10#820010)