[GitHub] flink pull request: [FLINK-3655] Multiple File Paths for InputFile...
Github user gna-phetsarath commented on the pull request: https://github.com/apache/flink/pull/1990#issuecomment-221890183 You are correct, the majority of the changes were in the "generate splits" method and "statistics" methods which included changes to subclasses that used the file path directly. Not as extensive as it appears. Also, additional tests were added. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3655) Allow comma-separated or multiple directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289131#comment-15289131
]
Gna Phetsarath commented on FLINK-3655:
---
There's a pull request for this: https://github.com/apache/flink/pull/1990
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.0.0
> Reporter: Gna Phetsarath
>Priority: Minor
> Labels: starter
>
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat so that a DataSource will process the directories
> sequentially.
>
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
> in Scala
>env.readFile(paths: Seq[String])
> or
> env.readFile(path: String, otherPaths: String*)
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3655] Multiple File Paths for InputFile...
GitHub user gna-phetsarath opened a pull request:
https://github.com/apache/flink/pull/1990
[FLINK-3655] Multiple File Paths for InputFileFormat.
I had to create a new PR, because I messed up my branches.
This addresses [FLINK-3655] Multiple File Paths for InputFileFormat but
does not implement file name globbing.
Also, this branch does not use guava.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-3655]
Multiple File Paths for InputFileFormat.")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
Removed Guava.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gna-phetsarath/flink
FLINK-3655-mulitple_directories_for_FileInputFormat_2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1990.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1990
commit 435a48339d0730714c90f61cfc4d435425e159e7
Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com>
Date: 2016-05-13T16:24:46Z
[FLINK-3655] Multiple File Paths for InputFileFormat.
Removed Guava.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-3655] Multiple File Paths for InputFile...
Github user gna-phetsarath closed the pull request at: https://github.com/apache/flink/pull/1987 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3655] Multiple File Paths for InputFile...
Github user gna-phetsarath commented on the pull request: https://github.com/apache/flink/pull/1987#issuecomment-219033497 I'll remove Guava later today or early tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3655] Multiple File Paths for InputFile...
GitHub user gna-phetsarath opened a pull request: https://github.com/apache/flink/pull/1987 [FLINK-3655] Multiple File Paths for InputFileFormat. This is the fix for: [FLINK-3655] Multiple File Paths for InputFileFormat. Does not support path Glob'ing. Will create another ticket for that. Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [ ] General - The pull request references the related JIRA issue ([FLINK-3655] Multiple File Paths for InputFileFormat.) - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [ ] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [ ] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed Added more unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gna-phetsarath/flink FLINK-3655-multiple_directories_for_FileInputFormat_1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1987.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1987 commit fb8918becfaae4dbb95aff2546364d83a0050494 Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com> Date: 2016-05-13T00:20:24Z [FLINK-3655] Multiple File Paths for InputFileFormat. Added more unit tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3655] multiple directories (Paths) for ...
Github user gna-phetsarath closed the pull request at: https://github.com/apache/flink/pull/1986 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3655] multiple directories (Paths) for ...
GitHub user gna-phetsarath opened a pull request:
https://github.com/apache/flink/pull/1986
[FLINK-3655] multiple directories (Paths) for FileInputFormat
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gna-phetsarath/flink
FLINK-3655-multiple_directories_for_FileInputFormat
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1986.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1986
commit 4908340f33c3e7d9541fb9b22025a9ae6dfafcbf
Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com>
Date: 2016-05-11T21:43:46Z
[FLINK-3665] Multiple directory support for FileInputFormat.
commit c35453e4765399885782b763e451fc7846842ed5
Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com>
Date: 2016-05-12T19:00:16Z
[FLINK-3655] Multiple File Paths for InputFileFormat.
Added more unit tests.
commit 9e19f8d5c616598b16cd6f2cf1f2d43df4155f74
Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com>
Date: 2016-05-12T20:47:26Z
Squash commit
commit da72c561b9c76558a04d5e467a0ce3c6f0b10366
Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com>
Date: 2016-05-12T20:48:18Z
Merge branch 'FLINK-3655-multiple_directories_for_FileInputFormat' of
github.com:gna-phetsarath/flink into
FLINK-3655-multiple_directories_for_FileInputFormat
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-3655) Allow comma-separated or multiple directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271314#comment-15271314
]
Gna Phetsarath commented on FLINK-3655:
---
What's the progress on this ticket, [~tianli]?
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.0.0
> Reporter: Gna Phetsarath
>Priority: Minor
> Labels: starter
>
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat so that a DataSource will process the directories
> sequentially.
>
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
> in Scala
>env.readFile(paths: Seq[String])
> or
> env.readFile(path: String, otherPaths: String*)
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [Flink-3691] extend avroinputformat to support...
Github user gna-phetsarath commented on a diff in the pull request:
https://github.com/apache/flink/pull/1920#discussion_r61957444
--- Diff:
flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/AvroRecordInputFormatTest.java
---
@@ -289,6 +290,119 @@ public void testDeserializeToSpecificType() throws
IOException {
}
}
+ /**
+* Test if the AvroInputFormat is able to properly read data from an
Avro
+* file as a GenericRecord.
+*
+* @throws IOException,
+* if there is an exception
+*/
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testDeserialisationGenericRecord() throws IOException {
+ Configuration parameters = new Configuration();
+
+ AvroInputFormat format = new
AvroInputFormat(new Path(testFile.getAbsolutePath()),
+ GenericRecord.class);
+ try {
+ format.configure(parameters);
+ FileInputSplit[] splits = format.createInputSplits(1);
+ assertEquals(splits.length, 1);
+ format.open(splits[0]);
+
+ GenericRecord u = format.nextRecord(null);
--- End diff --
From ```GenericData.class```, if you pass a null, a new instance of the
```Record``` will be created:
```
/**
* Called to create new record instances. Subclasses may override to use a
* different record implementation. The returned instance must conform to
the
* schema provided. If the old object contains fields not present in the
* schema, they should either be removed from the old object, or it should
* create a new instance that conforms to the schema. By default, this
returns
* a {@link GenericData.Record}.
*/
public Object newRecord(Object old, Schema schema) {
if (old instanceof IndexedRecord) {
IndexedRecord record = (IndexedRecord)old;
if (record.getSchema() == schema)
return record;
}
return new GenericData.Record(schema);
}
```
So, I think it's valid.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3691] extend avroinputformat to support...
Github user gna-phetsarath commented on the pull request: https://github.com/apache/flink/pull/1920#issuecomment-215887300 Many contributions to Flink is a great problem to heave. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [Flink-3691] extend avroinputformat to support...
Github user gna-phetsarath commented on the pull request: https://github.com/apache/flink/pull/1920#issuecomment-215716418 @greghogan - What are next steps? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [Flink-3691] extend avroinputformat to support...
Github user gna-phetsarath commented on a diff in the pull request:
https://github.com/apache/flink/pull/1920#discussion_r61269058
--- Diff:
flink-batch-connectors/flink-avro/src/test/java/org/apache/flink/api/io/avro/AvroRecordInputFormatTest.java
---
@@ -289,6 +290,119 @@ public void testDeserializeToSpecificType() throws
IOException {
}
}
+ /**
+* Test if the AvroInputFormat is able to properly read data from an
Avro
+* file as a GenericRecord.
+*
+* @throws IOException,
+* if there is an exception
+*/
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testDeserialisationGenericRecord() throws IOException {
+ Configuration parameters = new Configuration();
+
+ AvroInputFormat format = new
AvroInputFormat(new Path(testFile.getAbsolutePath()),
+ GenericRecord.class);
+ try {
+ format.configure(parameters);
+ FileInputSplit[] splits = format.createInputSplits(1);
+ assertEquals(splits.length, 1);
+ format.open(splits[0]);
+
+ GenericRecord u = format.nextRecord(null);
--- End diff --
```AvroInputFormat``` does not have a ```nextRecord()``` method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [Flink-3691] extend avroinputformat to support...
Github user gna-phetsarath commented on the pull request: https://github.com/apache/flink/pull/1920#issuecomment-215092007 ReflectDatumReader does not work with GenericRecord because it is an interface, so you need to use GenericDataReader. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [Flink-3691] extend avroinputformat to support...
Github user gna-phetsarath commented on a diff in the pull request:
https://github.com/apache/flink/pull/1920#discussion_r61261060
--- Diff:
flink-batch-connectors/flink-avro/src/main/java/org/apache/flink/api/java/io/AvroInputFormat.java
---
@@ -119,12 +144,18 @@ public E nextRecord(E reuseValue) throws IOException {
if (reachedEnd()) {
return null;
}
-
- if (!reuseAvroValue) {
- reuseValue =
InstantiationUtil.instantiate(avroValueType, Object.class);
+ if (isGenericRecord) {
--- End diff --
OK. I'll make the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Flink 3691 extend avroinputformat to support g...
GitHub user gna-phetsarath opened a pull request: https://github.com/apache/flink/pull/1920 Flink 3691 extend avroinputformat to support generic records Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [ ] General - The pull request references the related JIRA issue - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message - [ ] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [ ] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/gna-phetsarath/flink FLINK-3691-extend_avroinputformat_to_support_generic_records Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1920.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1920 commit 78b16a080105a188fcfc0f2a1731b87857f4080f Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com> Date: 2016-04-05T22:01:24Z [FLINK-3691] Extend AvroInputFormat to support Avro GenericRecord commit d122c6b0e125af39163514d286fe7abdbf16765d Author: Phetsarath, Sourigna <gna.phetsar...@teamaol.com> Date: 2016-04-06T19:21:06Z [FLINK-3691] Extend AvroInputFormat to support Avro GenericRecord. Fixed Style issue after running verify. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3655) Allow comma-separated or multiple directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226676#comment-15226676
]
Gna Phetsarath commented on FLINK-3655:
---
Will do be doing wildcards as well, or should be put that as another ticket?
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.0.0
> Reporter: Gna Phetsarath
>Priority: Minor
> Labels: starter
>
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat so that a DataSource will process the directories
> sequentially.
>
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
> in Scala
>env.readFile(paths: Seq[String])
> or
> env.readFile(path: String, otherPaths: String*)
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3691) Extend AvroInputFormat to support Avro GenericRecord
[ https://issues.apache.org/jira/browse/FLINK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gna Phetsarath updated FLINK-3691: -- Priority: Minor (was: Major) Component/s: Core I already have a working version, I need to create a branch and then a PR with all of the necessary tests. Should be able to get to this next week, if there are no objections. > Extend AvroInputFormat to support Avro GenericRecord > > > Key: FLINK-3691 > URL: https://issues.apache.org/jira/browse/FLINK-3691 > Project: Flink > Issue Type: Improvement > Components: Core > Reporter: Gna Phetsarath >Priority: Minor > > Extend AvroInputFormat to support Avro GenericRecord: > Sometimes the calling application will have access to the schema but not the > implementing class for Avro files. > Using a GenericDatumReader will allow the AvroInput format to read records in > the Avro Generic Record Format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3691) Extend AvroInputFormat to support Avro GenericRecord
Gna Phetsarath created FLINK-3691: - Summary: Extend AvroInputFormat to support Avro GenericRecord Key: FLINK-3691 URL: https://issues.apache.org/jira/browse/FLINK-3691 Project: Flink Issue Type: Improvement Reporter: Gna Phetsarath Extend AvroInputFormat to support Avro GenericRecord: Sometimes the calling application will have access to the schema but not the implementing class for Avro files. Using a GenericDatumReader will allow the AvroInput format to read records in the Avro Generic Record Format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-3655) Allow comma-separated or multiple directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gna Phetsarath updated FLINK-3655:
--
Description:
Allow comma-separated or multiple directories to be specified for
FileInputFormat so that a DataSource will process the directories sequentially.
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
in Scala
env.readFile(paths: Seq[String])
or
env.readFile(path: String, otherPaths: String*)
Wildcard support would be a bonus.
was:
Allow comma-separated directories to be specified for FileInputFormat so that a
DataSource will process the directories sequentially.
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
in Scala
env.readFile(paths: Seq[String])
or
env.readFile(path: String, otherPaths: String*)
Wildcard support would be a bonus.
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
>Reporter: Gna Phetsarath
>Priority: Minor
>
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat so that a DataSource will process the directories
> sequentially.
>
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
> in Scala
>env.readFile(paths: Seq[String])
> or
> env.readFile(path: String, otherPaths: String*)
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3655) Allow comma-separated or multiple directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gna Phetsarath updated FLINK-3655:
--
Summary: Allow comma-separated or multiple directories to be specified for
FileInputFormat (was: Allow comma-separated directories to be specified for
FileInputFormat)
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.0.0
> Reporter: Gna Phetsarath
>Priority: Minor
>
> Allow comma-separated directories to be specified for FileInputFormat so that
> a DataSource will process the directories sequentially.
>
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
> in Scala
>env.readFile(paths: Seq[String])
> or
> env.readFile(path: String, otherPaths: String*)
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3655) Allow comma-separated directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gna Phetsarath updated FLINK-3655:
--
Description:
Allow comma-separated directories to be specified for FileInputFormat so that a
DataSource will process the directories sequentially.
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
in Scala
env.readFile(paths: Seq[String])
or
env.readFile(path: String, otherPaths: String*)
Wildcard support would be a bonus.
was:
Allow comma-separated directories to be specified for FileInputFormat so that a
DataSource will process the directories sequentially.
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")}}
Wildcard support would be a bonus.
> Allow comma-separated directories to be specified for FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.0.0
>Reporter: Gna Phetsarath
>Priority: Minor
>
> Allow comma-separated directories to be specified for FileInputFormat so that
> a DataSource will process the directories sequentially.
>
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
> in Scala
>env.readFile(paths: Seq[String])
> or
> env.readFile(path: String, otherPaths: String*)
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3655) Allow comma-separated directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gna Phetsarath updated FLINK-3655:
--
Description:
Allow comma-separated directories to be specified for FileInputFormat so that a
DataSource will process the directories sequentially.
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")}}
Wildcard support would be a bonus.
was:
Allow comma-separated multiple directories to be specified for FileInputFormat.
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")}}
Wildcard support would be a bonus.
> Allow comma-separated directories to be specified for FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.0.0
>Reporter: Gna Phetsarath
>Priority: Minor
>
> Allow comma-separated directories to be specified for FileInputFormat so that
> a DataSource will process the directories sequentially.
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")}}
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-3655) Allow comma-separated directories to be specified for FileInputFormat
[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gna Phetsarath updated FLINK-3655:
--
Summary: Allow comma-separated directories to be specified for
FileInputFormat (was: Allow comma-separated multiple directories to be
specified for FileInputFormat)
> Allow comma-separated directories to be specified for FileInputFormat
> -
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 1.0.0
> Reporter: Gna Phetsarath
>Priority: Minor
>
> Allow comma-separated multiple directories to be specified for
> FileInputFormat.
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")}}
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3655) Allow comma-separated multiple directories to be specified for FileInputFormat
Gna Phetsarath created FLINK-3655:
-
Summary: Allow comma-separated multiple directories to be
specified for FileInputFormat
Key: FLINK-3655
URL: https://issues.apache.org/jira/browse/FLINK-3655
Project: Flink
Issue Type: Improvement
Components: Core
Affects Versions: 1.0.0
Reporter: Gna Phetsarath
Priority: Minor
Allow comma-separated multiple directories to be specified for FileInputFormat.
env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")}}
Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
