[ 
https://issues.apache.org/jira/browse/PARQUET-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689729#comment-17689729
 ] 

ASF GitHub Bot commented on PARQUET-2244:
-----------------------------------------

zhongyujiang commented on PR #1028:
URL: https://github.com/apache/parquet-mr/pull/1028#issuecomment-1433005114

   >  I don't know if there is a downstream that relies on Parquet judge value 
<> null as TRUE instead of UNKNOW, I guess that might be in some non-ansi 
standard engines.
   
   I just want to make sure it's safe to revert this, I did some search and 
find this: 
https://stackoverflow.com/questions/129077/null-values-inside-not-in-clause
   Seems SQL Server 2005 has a switch `ansi_null` to control wheather the 
result of  `value <> null` is `TRUE` or `UNKNOWN`,  when `ansi_nulls` is off, 
`3 <> null` is true. Though I didn't find such a switch in engines like Hive or 
Spark. Do you have any comments on this? Or do you think it's worth caring 
about?  @gszadovszky @wgtmac @huaxingao.




> Dictionary filter may skip row-groups incorrectly when evaluating notIn
> -----------------------------------------------------------------------
>
>                 Key: PARQUET-2244
>                 URL: https://issues.apache.org/jira/browse/PARQUET-2244
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-mr
>    Affects Versions: 1.12.2
>            Reporter: Yujiang Zhong
>            Assignee: Yujiang Zhong
>            Priority: Major
>
> Dictionary filter may skip row-groups incorrectly when evaluating `notIn` on 
> optional columns with null values. Here is an example:
> Say there is a optional column `c1` with all pages dict encoded, `c1` has and 
> only has two distinct values: ['foo', null],  and the predicate is  `c1 not 
> in ('foo', 'bar')`. 
> Now dictionary filter may skip this row-group that is actually should not be 
> skipped, because there are nulls in the column.
>  
> This is a bug similar to #1510.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to