ASF GitHub Bot commented on ORC-313:

GitHub user stiga-huang opened a pull request:


    ORC-313: Check subtype count of LIST, MAP and UNION types

    We need to verify the subtype count of LIST, MAP and UNION types. 
Otherwise, the ill types will lead the c++ reader to crash. The java reader 
also has this problem.
    Files attached to the JIRA can reproduce the crash.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/stiga-huang/orc check-subtype-cnt

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #229
commit 8661fc43ef76030cfb5d67844a7e6a3c26167cda
Author: stiga-huang <huangquanlong@...>
Date:   2018-03-03T09:21:23Z

    ORC-313: Check subtype count in converting protobuf Types
    We need to verify the subtype count of LIST, MAP and UNION types.
    Otherwise, the ill types will lead the c++ reader to crash.
    The java reader also has this problem.

commit 023f124550fc843d0d1aa0fe4d5c9e27e4d3bea9
Author: stiga-huang <huangquanlong@...>
Date:   2018-03-12T12:57:38Z

    add unit tests


> Missing checks on SubType count of LIST, MAP and UNION types
> ------------------------------------------------------------
>                 Key: ORC-313
>                 URL: https://issues.apache.org/jira/browse/ORC-313
>             Project: ORC
>          Issue Type: Bug
>          Components: C++, Java, Reader
>    Affects Versions: 1.0.0, 1.1.2, 1.2.3, 1.3.4, 1.4.3
>            Reporter: Quanlong Huang
>            Priority: Major
>         Attachments: ill_types.orc, no_subtypes_union.orc
> When converting proto::Type to TypeImpl, we need to check that LIST and MAP 
> types have a correct number of subtypes. Otherwise, it will lead to later 
> errors in the c++ reader.
> A file with ill types is attached.
> The java reader also has this problem and throws an IndexOutOfBoundsException 
> immediately:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0
>         at java.util.Collections$EmptyList.get(Collections.java:3212)
>         at org.apache.orc.OrcProto$Type.getSubtypes(OrcProto.java:12642)
>         at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:506)
>         at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:515)
>         at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:386)
>         at org.apache.orc.OrcFile.createReader(OrcFile.java:327)
>         at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
>         at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:300)
>         at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
>         at org.apache.orc.tools.FileDump.main(FileDump.java:135)
>         at org.apache.orc.tools.Driver.main(Driver.java:105)
> {code}
> We should also check the subtype count of UNION type. If its subtype_size == 
> 0, the 'counts' pointer used in UnionColumnReader::skip and 
> UnionColumnReader::next will be null pointer. The attached file 
> (no_subtypes_union.orc) can reproduce this.

This message was sent by Atlassian JIRA

Reply via email to