Jacob Tolar created PIG-5432:
--------------------------------
Summary: OrcStorage fails to detect schema in some cases
Key: PIG-5432
URL: https://issues.apache.org/jira/browse/PIG-5432
Project: Pig
Issue Type: New Feature
Reporter: Jacob Tolar
OrcStorage needs to detect the schema of input data paths. If some data paths
have no files this will fail.
For example:
{code}
A = LOAD '/path/to/20230101,/path/to/20230102' USING OrcStorage();
{code}
If {{/path/to/20230101}} contains only a _SUCCESS marker and {{20230102}}
contains data, OrcStorage fails to detect the schema.
The code tries to use a search algorithm to recursively search through all
input paths for the data (via Utils.depthFirstSearchForFile), but it is
implemented incorrectly and returns early in this scenario.
See:
https://github.com/apache/pig/blob/c0d75ba930f9aa5c6454d0264a96f82b45279202/src/org/apache/pig/builtin/OrcStorage.java#L389-L408
https://github.com/apache/pig/blob/59ec4a326079c9f937a052194405415b1e3a2b06/src/org/apache/pig/impl/util/Utils.java#L629-L667
I'll attach a patch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
