GitHub user swegner opened a pull request:
https://github.com/apache/incubator-beam/pull/1627
[BEAM-1160] Add option to disable failures if filePattern resolves to empty
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.txt).
---
Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.
Most PTransforms also have an option to disable their
construction-time validation. This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.
This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/swegner/incubator-beam read-allow-empty-glob
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1627.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1627
----
commit 43195ba27b4303c852e81fa1de493ec4fec641bb
Author: Scott Wegner <sweg...@google.com>
Date: 2016-12-15T18:12:25Z
Add option to disable failures if filePattern resolves to empty
Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.
Most PTransforms also have an option to disable their
construction-time validation. This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.
This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
