Required fields projection in Loader: nested fields in bag, map key lookup more
than two levels
-----------------------------------------------------------------------------------------------
Key: PIG-932
URL: https://issues.apache.org/jira/browse/PIG-932
Project: Pig
Issue Type: Improvement
Components: impl
Affects Versions: 0.3.0
Reporter: Daniel Dai
To leverage the performance features provided by Zebra, Pig should be able to
figure out which input fields are actually used in Pig script, and prune
unnecessary inputs. This feature is being implementing in
[PIG-922|https://issues.apache.org/jira/browse/PIG-922]. However, there are two
limitations currently:
1. Pruning nested fields only apply to map and record. We do not prune
sub-field inside a collection
2. For map, currently we only go one level deep. Eg, if in Pig script, user
uses a#'key0'#'key1', a#'key0' will be asked
These two limitations are in line with current limitation of Zebra loader. Once
Zebra loader can handle this, we need to work to lift these limitations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
