airajena commented on code in PR #102:
URL:
https://github.com/apache/fineract-business-intelligence/pull/102#discussion_r4318504210
##########
contracts/fineract_tables.json:
##########
@@ -0,0 +1,320 @@
+{
+ "_license": "Licensed to the Apache Software Foundation (ASF) under one or
more contributor license agreements. See the NOTICE file distributed with this
work for additional information regarding copyright ownership. The ASF licenses
this file to You under the Apache License, Version 2.0 (the \"License\"); you
may not use this file except in compliance with the License. You may obtain a
copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless
required by applicable law or agreed to in writing, software distributed under
the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.",
+ "_doc": "Single source of truth for every ingested Fineract table. Read by
the warehouse DDL provisioner (meta.provision_raw_tables), the Python
extractor, the Java CDC sink, the Debezium connector builder and the load
generator. Column 'cdc' lists base-table column candidates (first present
wins); 'synthetic' columns are written as NULL by every engine so both engines
produce identical raw rows.",
+ "contract_version": 2,
+ "topic_prefix": "fineract.cdc",
+ "source_schema": "public",
+ "tables": [
+ {
+ "table_name": "m_office",
+ "raw_table": "raw_m_office",
+ "primary_key": "id",
+ "python_cursor": null,
+ "kafka_partitions": 3,
+ "change_tracking": {"entity": "global"},
+ "columns": [
+ {"name": "id", "type": "bigint"},
+ {"name": "parent_id", "type": "bigint"},
+ {"name": "hierarchy", "type": "text"},
+ {"name": "external_id", "type": "text"},
+ {"name": "name", "type": "text"},
+ {"name": "opening_date", "type": "date"},
+ {"name": "created_on_utc", "type": "timestamptz", "synthetic": true},
+ {"name": "last_modified_on_utc", "type": "timestamptz", "synthetic":
true}
+ ]
+ },
+ {
+ "table_name": "m_currency",
+ "raw_table": "raw_m_currency",
+ "primary_key": "id",
+ "python_cursor": null,
+ "kafka_partitions": 1,
+ "change_tracking": null,
+ "columns": [
+ {"name": "id", "type": "bigint"},
+ {"name": "code", "type": "text"},
+ {"name": "decimal_places", "type": "smallint"},
+ {"name": "currency_multiplesof", "type": "smallint"},
+ {"name": "display_symbol", "type": "text"},
+ {"name": "name", "type": "text"},
+ {"name": "internationalized_name_code", "type": "text"},
+ {"name": "created_on_utc", "type": "timestamptz", "synthetic": true},
+ {"name": "last_modified_on_utc", "type": "timestamptz", "synthetic":
true}
+ ]
+ },
+ {
+ "table_name": "m_client",
+ "raw_table": "raw_m_client",
+ "primary_key": "id",
+ "python_cursor": "last_modified_on_utc",
+ "kafka_partitions": 12,
+ "change_tracking": {"entity": "client", "key": "id", "date": null},
+ "columns": [
+ {"name": "id", "type": "bigint"},
+ {"name": "account_no", "type": "text"},
+ {"name": "external_id", "type": "text"},
+ {"name": "status_enum", "type": "integer"},
+ {"name": "activation_date", "type": "date"},
+ {"name": "office_joining_date", "type": "date"},
+ {"name": "office_id", "type": "bigint"},
+ {"name": "staff_id", "type": "bigint"},
+ {"name": "gender_cv_id", "type": "integer"},
+ {"name": "date_of_birth", "type": "date", "pii": true},
Review Comment:
Fixed. Columns flagged pii are excluded from the Debezium column list and
the Python reads by default (FBI_EXCLUDE_PII=true). client_hash is now an HMAC
keyed by FBI_PII_HASH_KEY, kept outside the warehouse. A source already loaded
with date_of_birth keeps those values in raw until it is re-backfilled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
