[jira] [Commented] (FLINK-8243) OrcTableSource should recursively read all files in nested directories of the input path.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352600#comment-16352600
 ] 

ASF GitHub Bot commented on FLINK-8243:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5344


> OrcTableSource should recursively read all files in nested directories of the 
> input path.
> -
>
> Key: FLINK-8243
> URL: https://issues.apache.org/jira/browse/FLINK-8243
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> The {{OrcTableSource}} only reads files on the first level of the provided 
> input path. 
> Hive's default behavior is to recursively read all nested files in the input 
> path. We should follow this behavior and add a switch to disable recursive 
> reading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8243) OrcTableSource should recursively read all files in nested directories of the input path.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352233#comment-16352233
 ] 

ASF GitHub Bot commented on FLINK-8243:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/5344
  
Merging


> OrcTableSource should recursively read all files in nested directories of the 
> input path.
> -
>
> Key: FLINK-8243
> URL: https://issues.apache.org/jira/browse/FLINK-8243
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> The {{OrcTableSource}} only reads files on the first level of the provided 
> input path. 
> Hive's default behavior is to recursively read all nested files in the input 
> path. We should follow this behavior and add a switch to disable recursive 
> reading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8243) OrcTableSource should recursively read all files in nested directories of the input path.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336365#comment-16336365
 ] 

ASF GitHub Bot commented on FLINK-8243:
---

GitHub user fhueske opened a pull request:

https://github.com/apache/flink/pull/5344

[FLINK-8243] [orc] OrcTableSource reads input path recursively by default.

## What is the purpose of the change

- enable recursive directory enumeration for `OrcTableSource` (Hive's 
default behavior)
- expose the configuration in the `OrcTableSource` builder.

## Brief change log

* see purpose above

## Verifying this change

* Unit test to validate configuration of `FileInputFormat` has been added 
in `OrcTableSourceTest.testBuilder()`

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): **no**
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: **no**
  - The serializers: **no**
  - The runtime per-record code paths (performance sensitive): **no**
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
  - The S3 file system connector: **no**

## Documentation

  - Does this pull request introduce a new feature? **yes**
  - If yes, how is the feature documented? Java docs of builder + web 
documentation.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhueske/flink orcRecursive

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5344.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5344


commit e8ff9948d16bf4f0884644ba123f067e697e5a73
Author: Fabian Hueske 
Date:   2017-12-12T11:18:23Z

[FLINK-8243] [orc] OrcTableSource reads input path recursively by default.




> OrcTableSource should recursively read all files in nested directories of the 
> input path.
> -
>
> Key: FLINK-8243
> URL: https://issues.apache.org/jira/browse/FLINK-8243
> Project: Flink
>  Issue Type: Improvement
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>Priority: Critical
> Fix For: 1.5.0, 1.4.1
>
>
> The {{OrcTableSource}} only reads files on the first level of the provided 
> input path. 
> Hive's default behavior is to recursively read all nested files in the input 
> path. We should follow this behavior and add a switch to disable recursive 
> reading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)