[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2330 Merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2330 I disabled tests for the hadoop1 profile. Will build the PR one more time and merge if everything passes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2330 I'll propose to drop the hadoop1 builds on the dev ML. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2330 I noticed building this PR for hadoop1 (`mvn clean install -Dhadoop.profile=1`) fails: > The following artifacts could not be resolved: org.apache.hadoop:hadoop-hdfs:jar:tests:1.2.1, org.apache.hbase:hbase-hadoop2-compat:jar:tests:0.98.11-hadoop1: Could not find artifact org.apache.hadoop:hadoop-hdfs:jar:tests:1.2.1 in central I'm not a Maven guru. Is it possible to disable compiling and executing the tests for the hadoop1 profile? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2330 Hi @nielsbasjes, I just posted to the dev mailinglist and proposed to update the HBase dependency to 1.2.3 (as [FLINK-2765](https://issues.apache.org/jira/browse/FLINK-2765) suggests). By the end of the week we have a decision and I will merge this PR to the master branch. In the meantime, I will merge the fixed TableInputFormat changes to the Flink 1.1 branch and revert all breaking changes (pom.xml, RichInputFormat, hbase-site.xml, tests, ...). For Flink 1.2.0 we want these changes. Thanks, Fabian --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2330 Hi @nielsbasjes, thanks for fixing and cleaning up the `TableInputFormat`. This PR is good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 Current version has a problem in building the shaded jars. I runs into an infinite loop in creating the dependency-reduced-pom.xml as described here: **Shade Plugin gets stuck in infinite loop building dependency reduced POM** https://issues.apache.org/jira/browse/MSHADE-148 Although all my versions are newer than the fix described there I still see the problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 I managed to resolve the problems with running these unit tests. These problems were caused by version conflicts in guava. Now we have a HBaseMiniCluster that is started, a table with multiple regions is created. And the TableInputFormat is used to extract the rows again. By setting the paralellism to 1 the same TableInputFormat instance is used for multiple regions and succeeds (the problem this all started with). Please review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 I did a few serious attempts to create a unit test that fires the HBaseMiniCluster ... and failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 I will add a unit test for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 Question: Is this change good? Or do you have more things that I need to change before it can be committed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 I had another look at the "multiple tables" question. The name of the table comes from the getTableName method that is to be implemented by the subclass. I consider it to be extremely unlikely that multiple calls to that method in a single instance will yield different table names. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2330 I don't know, and it seems the InputFormat itself doesn't know either. If we go by the previous implementation then yes, there will only be one table. However, based on the comments on Line 64: `// abstract methods allow for multiple table and scanners in the same job` we have to conclude that there can be different tables. I'd be curious what @twalthr thinks about this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 Note that this version still assumes that the single instance will only see multiple splits for the same table. Is that a safe assumption? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 Now I see why I missed these two; They are newer than the 1.0.3 I was working with. Is it a good idea to add ' throws IOException' to these two in RichInputFormat ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2330 I would say yes, since `open()` and `close()` can also throw an `IOException`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 Yes, that is indeed the right place to do this. Bummer this method does not allow throwing exceptions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user zentol commented on the issue: https://github.com/apache/flink/pull/2330 maybe you can move the table initialization into `openInputFormat()` (called once before all splits) and close it in `closeInputFormat()` (called once after all splits). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2330: FLINK-4311 Fixed several problems in TableInputFormat
Github user nielsbasjes commented on the issue: https://github.com/apache/flink/pull/2330 Oh damn, I just noticed a major issue in this: In order to create the input splits the table needs to be available "before" the call to the 'open' method. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
