[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read
Github user neoremind commented on the issue: https://github.com/apache/flink/pull/5639 @fhueske Thanks for your response. I understand this case. Please take your time, hope this PR can be reviewed in the future and help people who needed. Thanks! ---
[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read
Github user neoremind commented on the issue: https://github.com/apache/flink/pull/5639 @ramkrish86 @fhueske could you help to review this PR? Since I noticed that you guys contribute most of the code. This update enables HBase snapshot read and I refactor some of the code and test cases , you can find design doc and class diagram on https://issues.apache.org/jira/projects/FLINK/issues/FLINK-8862?filter=allopenissues. Thanks! ---
[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read
Github user neoremind commented on the issue: https://github.com/apache/flink/pull/5639 @zentol I was thinking could you help me review the feature and code? Or is there anyone more appropriate for this PR? many thanks. ---
[GitHub] flink pull request #5639: [FLINK-8862] [HBase] Support HBase snapshot read
GitHub user neoremind opened a pull request: https://github.com/apache/flink/pull/5639 [FLINK-8862] [HBase] Support HBase snapshot read ## What is the purpose of the change *Flink-hbase connector only supports reading/scanning HBase over region server scanner, there is also [snapshot](http://hbase.apache.org/book.html#ops.snapshots) scanning solution, just like Hadoop provides 2 ways to scan HBase, one is [TableInputFormat](https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableInputFormat.html), the other is [TableSnapshotInputFormat](https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.html), so it would be great if flink supports both solutions to ensure more wider usage scope and provide alternatives for users.* ## Brief change log - *Create `TableInputSplitStrategy` interface and its implementations as abstraction logic for `AbstractTableInputFormat`* - *Update `HBaseRowInputFormat` and `TableInputFormat`* - *Add `HBaseSnapshotRowInputFormat` and `TableSnapshotInputFormat`* - *Extract 2 interfaces including `HBaseTableScannerAware` and `ResultToTupleMapper`* - *Add `HBaseSnapshotReadExample`* ## Verifying this change This change is already covered by existing tests as follows, and new test cases has been added as well. `org.apache.flink.addons.hbase.HBaseConnectorITCase` This change added tests and can be verified as follows: - *Manually create one snapshot for a specific HBase table, and use TableSnapshotInputFormat to do full scan.* - *Running existing HBaseReadExample to do full scan.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / **docs** / **JavaDocs** / not documented) - For document, please visit [JIRA ticket](https://issues.apache.org/jira/projects/FLINK/issues/FLINK-8862?filter=allopenissues), a detailed design doc and class diagram have been attached. You can merge this pull request into a Git repository by running: $ git pull https://github.com/neoremind/flink snapshot Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5639.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5639 commit 0b36b434f987a971b6463ce3441c483380cfa9dd Author: neoremind <xuzh1002@...> Date: 2018-03-05T14:14:09Z Support HBase snapshot read ---
[GitHub] flink pull request #5633: [FLINK-8857] [Hbase] Avoid HBase connector read ex...
Github user neoremind commented on a diff in the pull request:
https://github.com/apache/flink/pull/5633#discussion_r172125701
--- Diff:
flink-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseReadExample.java
---
@@ -86,8 +86,8 @@ public boolean filter(Tuple2<String, String> t) throws
Exception {
hbaseDs.print();
- // kick off execution.
- env.execute();
+ // kick off execution is not needed.
+ // env.execute();
--- End diff --
Done.
---
[GitHub] flink pull request #5633: [FLINK-8857] [Hbase] Avoid HBase connector read ex...
Github user neoremind commented on a diff in the pull request:
https://github.com/apache/flink/pull/5633#discussion_r172122738
--- Diff:
flink-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseReadExample.java
---
@@ -86,8 +86,8 @@ public boolean filter(Tuple2<String, String> t) throws
Exception {
hbaseDs.print();
- // kick off execution.
- env.execute();
+ // kick off execution is not needed.
+ // env.execute();
--- End diff --
Cool! So should I close this PR?
---
[GitHub] flink pull request #5633: [FLINK-8857] [Hbase] Avoid HBase connector read ex...
GitHub user neoremind opened a pull request: https://github.com/apache/flink/pull/5633 [FLINK-8857] [Hbase] Avoid HBase connector read example throwing exception at the end ## What is the purpose of the change *This pull request fixes problem of HBase read example throwing exception at the end of the program execution.* ## Brief change log - *Update example `flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseReadExample.java` by removing the part causing the problem.* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/neoremind/flink FLINK-8857 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5633.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5633 commit 97c009a4d2308ad1da5629f9653bed9af352a8f7 Author: neoremind <xuzh1002@...> Date: 2018-03-05T08:50:37Z Avoid hbase connector read example throwing exception at the end. ---
