GitHub user jackylk opened a pull request:
https://github.com/apache/incubator-carbondata/pull/208
[CARBONDATA-284][WIP] Abstracting index and segment interface
This PR adds new User API and Dev API for carbon-hadoop module:
### User API
- `CarbonColumnarInputFormat/OutputFormat`: it uses current
`CarbonInputFormat` as internal implementation.
- `CarbonRowInputFormat/OutputFormat`: it needs to be implemented
- `CarbonOutputCommitter`: used for managing segment commit
They are based on `CarbonInputFormatBase/OutputFormatBase`
### Dev API
- Segment: an abstract class represents a single load of data, used by
CarbonInputFormatBase to get all InputSplit by matching QueryModel, and used by
CarbonOutputCommitter to prepare for reading. Implementation examples are
`IndexedSegment` and `StreamingSegment`.
- SegmentManager: an interface to manage segments. Current implementation
is `ZkSegmentManager`, which need to be mapped to existing logic.
- Index: an interface that can is used by `IndexedSegment` to filter
InputSplit. Current implementation is `InMemoryBTreeIndex` which load the index
into driver's memory.
`CarbonInputFormatUtil` is modified so that it can also be used by
`CarbonColumnarInputFormat`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata index-interface
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/208.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #208
----
commit 398d2ec3e6706c615918a734a90f9dc4111067d8
Author: jackylk <[email protected]>
Date: 2016-10-03T16:01:48Z
add User API
commit 1d92a00403faeebc09bf595ba11b3e55d4c997f2
Author: jackylk <[email protected]>
Date: 2016-10-03T16:02:04Z
add Developer API
commit 1812a0a68b53ba5d48fc030e2a59329b0e827b05
Author: jackylk <[email protected]>
Date: 2016-10-03T16:02:49Z
refactory existing code
commit 430e7710b88725b587c1f3542d4d66ab02958cbc
Author: jackylk <[email protected]>
Date: 2016-10-03T16:27:10Z
change Index interface
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
