[jira] [Comment Edited] (HIVE-18608) ORC should allow selectively disabling dictionary-encoding on specified columns
[
https://issues.apache.org/jira/browse/HIVE-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379551#comment-16379551
]
Owen O'Malley edited comment on HIVE-18608 at 2/28/18 12:26 AM:
I've just opened a jira and a pull request that is useful to this and other
changes that need to specify column names.
https://issues.apache.org/jira/browse/ORC-308
it allows you to specify subfields by name such as your example:
myInfoArray.\_elem.emailBody.
was (Author: owen.omalley):
I've just opened a jira and a pull request that is useful to this and other
changes that need to specify column names.
https://issues.apache.org/jira/browse/ORC-308
it allows you to specify subfields by name such as your example:
myInfoArray._elem_.emailBody.
> ORC should allow selectively disabling dictionary-encoding on specified
> columns
> ---
>
> Key: HIVE-18608
> URL: https://issues.apache.org/jira/browse/HIVE-18608
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Affects Versions: 3.0.0, 2.4.0, 2.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Major
> Attachments: HIVE-18608.1-branch-2.2.patch
>
>
> Just as ORC allows the choice of columns to enable bloom-filters on, it would
> be nice to have a way to specify which columns {{DICTIONARY_V2}} encoding
> should be disabled on.
> Currently, the choice of dictionary-encoding depends on the results of
> sampling the first row-stride within a stripe. If the user knows that a
> column's cardinality is bound to prevent an effective dictionary, she might
> choose to simply disable it on just that column, and avoid the cost of
> sampling in the first row-stride.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18608) ORC should allow selectively disabling dictionary-encoding on specified columns
[
https://issues.apache.org/jira/browse/HIVE-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349660#comment-16349660
]
Mithun Radhakrishnan edited comment on HIVE-18608 at 2/2/18 2:03 AM:
-
I've attached an initial implementation, where dictionary encoding might be
disabled via a table-property ({{'orc.skip.dictionary.for.columns'}}.
Note: I've only added support for top-level columns. Specifying this on a
{{STRUCT}} will disable dictionary encoding for the entire sub-tree (i.e. all
members of the STRUCT), recursively.
It might be good to support selection at an arbitrary depth, at a later date.
E.g. {{myInfoArray.__elem__.emailBody}}.
was (Author: mithun):
I've attached an initial implementation, where dictionary encoding might be
disabled via a table-property ({{'orc.skip.dictionary.for.columns'}}.
Note: I've only added support for top-level columns. Specifying this on a
{{STRUCT}} will disable dictionary encoding for the entire sub-tree (i.e. all
members of the STRUCT), recursively.
It might be good to support selection at an arbitrary depth. E.g.
{{myInfoArray.__elem__.emailBody}}.
> ORC should allow selectively disabling dictionary-encoding on specified
> columns
> ---
>
> Key: HIVE-18608
> URL: https://issues.apache.org/jira/browse/HIVE-18608
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Affects Versions: 3.0.0, 2.4.0, 2.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Major
> Attachments: HIVE-18608.1-branch-2.2.patch
>
>
> Just as ORC allows the choice of columns to enable bloom-filters on, it would
> be nice to have a way to specify which columns {{DICTIONARY_V2}} encoding
> should be disabled on.
> Currently, the choice of dictionary-encoding depends on the results of
> sampling the first row-stride within a stripe. If the user knows that a
> column's cardinality is bound to prevent an effective dictionary, she might
> choose to simply disable it on just that column, and avoid the cost of
> sampling in the first row-stride.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18608) ORC should allow selectively disabling dictionary-encoding on specified columns
[
https://issues.apache.org/jira/browse/HIVE-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349660#comment-16349660
]
Mithun Radhakrishnan edited comment on HIVE-18608 at 2/2/18 2:03 AM:
-
I've attached an initial implementation, where dictionary encoding might be
disabled via a table-property ({{'orc.skip.dictionary.for.columns'}}).
Note: I've only added support for top-level columns. Specifying this on a
{{STRUCT}} will disable dictionary encoding for the entire sub-tree (i.e. all
members of the STRUCT), recursively.
It might be good to support selection at an arbitrary depth, at a later date.
E.g. {{myInfoArray.__elem__.emailBody}}.
was (Author: mithun):
I've attached an initial implementation, where dictionary encoding might be
disabled via a table-property ({{'orc.skip.dictionary.for.columns'}}.
Note: I've only added support for top-level columns. Specifying this on a
{{STRUCT}} will disable dictionary encoding for the entire sub-tree (i.e. all
members of the STRUCT), recursively.
It might be good to support selection at an arbitrary depth, at a later date.
E.g. {{myInfoArray.__elem__.emailBody}}.
> ORC should allow selectively disabling dictionary-encoding on specified
> columns
> ---
>
> Key: HIVE-18608
> URL: https://issues.apache.org/jira/browse/HIVE-18608
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Affects Versions: 3.0.0, 2.4.0, 2.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Major
> Attachments: HIVE-18608.1-branch-2.2.patch
>
>
> Just as ORC allows the choice of columns to enable bloom-filters on, it would
> be nice to have a way to specify which columns {{DICTIONARY_V2}} encoding
> should be disabled on.
> Currently, the choice of dictionary-encoding depends on the results of
> sampling the first row-stride within a stripe. If the user knows that a
> column's cardinality is bound to prevent an effective dictionary, she might
> choose to simply disable it on just that column, and avoid the cost of
> sampling in the first row-stride.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18608) ORC should allow selectively disabling dictionary-encoding on specified columns
[
https://issues.apache.org/jira/browse/HIVE-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349660#comment-16349660
]
Mithun Radhakrishnan edited comment on HIVE-18608 at 2/2/18 2:03 AM:
-
I've attached an initial implementation, where dictionary encoding might be
disabled via a table-property ({{'orc.skip.dictionary.for.columns'}}).
Note: I've only added support for top-level columns. Specifying this on a
{{STRUCT}} will disable dictionary encoding for the entire sub-tree (i.e. all
members of the {{STRUCT}}, recursively).
It might be good to support selection at an arbitrary depth, at a later date.
E.g. {{myInfoArray.__elem__.emailBody}}.
was (Author: mithun):
I've attached an initial implementation, where dictionary encoding might be
disabled via a table-property ({{'orc.skip.dictionary.for.columns'}}).
Note: I've only added support for top-level columns. Specifying this on a
{{STRUCT}} will disable dictionary encoding for the entire sub-tree (i.e. all
members of the STRUCT), recursively.
It might be good to support selection at an arbitrary depth, at a later date.
E.g. {{myInfoArray.__elem__.emailBody}}.
> ORC should allow selectively disabling dictionary-encoding on specified
> columns
> ---
>
> Key: HIVE-18608
> URL: https://issues.apache.org/jira/browse/HIVE-18608
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Affects Versions: 3.0.0, 2.4.0, 2.2.1
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>Priority: Major
> Attachments: HIVE-18608.1-branch-2.2.patch
>
>
> Just as ORC allows the choice of columns to enable bloom-filters on, it would
> be nice to have a way to specify which columns {{DICTIONARY_V2}} encoding
> should be disabled on.
> Currently, the choice of dictionary-encoding depends on the results of
> sampling the first row-stride within a stripe. If the user knows that a
> column's cardinality is bound to prevent an effective dictionary, she might
> choose to simply disable it on just that column, and avoid the cost of
> sampling in the first row-stride.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
