[
https://issues.apache.org/jira/browse/ATLAS-4153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dharshana M Krishnamoorthy updated ATLAS-4153:
----------------------------------------------
Description:
*Steps to re-produce:*
* Stop kafka broker
* Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
abc_table_n)
* Make sure the data is spooled
* Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
xyz_table_n)
* Wait for 5 mins for the tables to reflect in atlas
* Fire a basic search with abc_* prefix and verify the tables are created
* Collect the createdTime of all the table and verify the order
Here tables were created with prefix btwxb_table.
Guid: a55ca87c-1122-4ed4-9812-5047cc8a7968 of *btwxb_table_6* has "createTime":
*1612959902106*,
{code:java}
{
"referredEntities": {...},
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "hrt_qa",
"temporary": false,
"lastAccessTime": 1612959527000,
"aliases": null,
"replicatedTo": null,
"userDescription": null,
"replicatedFrom": null,
"qualifiedName": "default.btwxb_table_6@cm",
"displayName": null,
"columns": [
{
"guid": "3cf7b33b-7fb7-4de4-a967-7f89410b3a00",
"typeName": "hive_column"
},
{
"guid": "7c5b72ae-e100-467a-b3b2-caade0b7a7a1",
"typeName": "hive_column"
}
],
"description": null,
"viewExpandedText": null,
"tableType": "MANAGED_TABLE",
"sd": {
"guid": "7a1e2b04-f786-4f42-a3c4-31a0ad872256",
"typeName": "hive_storagedesc"
},
"createTime": 1612959527000,
"name": "btwxb_table_6",
"comment": null,
"partitionKeys": [],
"parameters": {...},
"retention": 0,
"viewOriginalText": null,
"db": {
"guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
"typeName": "hive_db"
}
},
"guid": "a55ca87c-1122-4ed4-9812-5047cc8a7968",
"isIncomplete": false,
"status": "ACTIVE",
"createdBy": "hive",
"updatedBy": "hrt_qa",
"createTime": 1612959902106,
"updateTime": 1612959911756,
"version": 0,
"relationshipAttributes": {...},
"labels": []
}
} {code}
Guid: 70738a7e-d29f-4234-82e7-adbdc4feaed6 of *btwxb_table_1* has "createTime":
*1612959903892*,
{code:java}
{
"referredEntities": {...},
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "hrt_qa",
"temporary": false,
"lastAccessTime": 1612959523000,
"aliases": null,
"replicatedTo": null,
"userDescription": null,
"replicatedFrom": null,
"qualifiedName": "default.btwxb_table_1@cm",
"displayName": null,
"columns": [
{
"guid": "c283ee8c-d292-406a-a062-d81b12614636",
"typeName": "hive_column"
},
{
"guid": "cbb19940-e513-4295-a145-5b2d6691ea13",
"typeName": "hive_column"
}
],
"description": null,
"viewExpandedText": null,
"tableType": "MANAGED_TABLE",
"sd": {
"guid": "69d7f6fc-8194-412f-a723-5b142ade1c31",
"typeName": "hive_storagedesc"
},
"createTime": 1612959523000,
"name": "btwxb_table_1",
"comment": null,
"partitionKeys": [],
"parameters": {...},
"retention": 0,
"viewOriginalText": null,
"db": {
"guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
"typeName": "hive_db"
}
},
"guid": "70738a7e-d29f-4234-82e7-adbdc4feaed6",
"isIncomplete": false,
"status": "ACTIVE",
"createdBy": "hive",
"updatedBy": "hrt_qa",
"createTime": 1612959903892,
"updateTime": 1612959919910,
"version": 0,
"relationshipAttributes": {...},
"labels": []
}
} {code}
Attributes's create time:
*btwxb_table_1 : 1612959523000*
*btwxb_table_6 : 1612959527000***
In hive *btwxb_table_1* is created before *btwxb_table_6*
Entity's create time:**
*btwxb_table_1* has "createTime": *1612959903892*, and
*btwxb_table_6* has "createTime": *1612959902106*, which is smaller than
*btwxb_table_1's* createTime.
*Which means table_6 is created before table_1 in atlas*
was:
*Steps to re-produce:*
* Stop kafka broker
* Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
abc_table_n)
* Make sure the data is spooled
* Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
xyz_table_n)
* Wait for 5 mins for the tables to reflect in atlas
* Fire a basic search with abc_* prefix and verify the tables are created
* Collect the createdTime of all the table and verify the order
Here tables were created with prefix btwxb_table.
Guid: a55ca87c-1122-4ed4-9812-5047cc8a7968 of *btwxb_table_6* has "createTime":
*1612959902106*,
{code:java}
{
"referredEntities": {...},
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "hrt_qa",
"temporary": false,
"lastAccessTime": 1612959527000,
"aliases": null,
"replicatedTo": null,
"userDescription": null,
"replicatedFrom": null,
"qualifiedName": "default.btwxb_table_6@cm",
"displayName": null,
"columns": [
{
"guid": "3cf7b33b-7fb7-4de4-a967-7f89410b3a00",
"typeName": "hive_column"
},
{
"guid": "7c5b72ae-e100-467a-b3b2-caade0b7a7a1",
"typeName": "hive_column"
}
],
"description": null,
"viewExpandedText": null,
"tableType": "MANAGED_TABLE",
"sd": {
"guid": "7a1e2b04-f786-4f42-a3c4-31a0ad872256",
"typeName": "hive_storagedesc"
},
"createTime": 1612959527000,
"name": "btwxb_table_6",
"comment": null,
"partitionKeys": [],
"parameters": {...},
"retention": 0,
"viewOriginalText": null,
"db": {
"guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
"typeName": "hive_db"
}
},
"guid": "a55ca87c-1122-4ed4-9812-5047cc8a7968",
"isIncomplete": false,
"status": "ACTIVE",
"createdBy": "hive",
"updatedBy": "hrt_qa",
"createTime": 1612959902106,
"updateTime": 1612959911756,
"version": 0,
"relationshipAttributes": {...},
"labels": []
}
} {code}
Guid: 70738a7e-d29f-4234-82e7-adbdc4feaed6 of *btwxb_table_1* has "createTime":
*1612959903892*,
{code:java}
{
"referredEntities": {...},
"entity": {
"typeName": "hive_table",
"attributes": {
"owner": "hrt_qa",
"temporary": false,
"lastAccessTime": 1612959523000,
"aliases": null,
"replicatedTo": null,
"userDescription": null,
"replicatedFrom": null,
"qualifiedName": "default.btwxb_table_1@cm",
"displayName": null,
"columns": [
{
"guid": "c283ee8c-d292-406a-a062-d81b12614636",
"typeName": "hive_column"
},
{
"guid": "cbb19940-e513-4295-a145-5b2d6691ea13",
"typeName": "hive_column"
}
],
"description": null,
"viewExpandedText": null,
"tableType": "MANAGED_TABLE",
"sd": {
"guid": "69d7f6fc-8194-412f-a723-5b142ade1c31",
"typeName": "hive_storagedesc"
},
"createTime": 1612959523000,
"name": "btwxb_table_1",
"comment": null,
"partitionKeys": [],
"parameters": {...},
"retention": 0,
"viewOriginalText": null,
"db": {
"guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
"typeName": "hive_db"
}
},
"guid": "70738a7e-d29f-4234-82e7-adbdc4feaed6",
"isIncomplete": false,
"status": "ACTIVE",
"createdBy": "hive",
"updatedBy": "hrt_qa",
"createTime": 1612959903892,
"updateTime": 1612959919910,
"version": 0,
"relationshipAttributes": {...},
"labels": []
}
} {code}
*btwxb_table_1* has "createTime": *1612959903892*, and
*btwxb_table_6* has "createTime": *1612959902106*, which is smaller than
*btwxb_table_1's* createTime.
Which means it is created before table_6 is created before table_1 in atlas
> [Atlas: Spooling] The order of the entities created in atlas is not same as
> the order created in hive
> -----------------------------------------------------------------------------------------------------
>
> Key: ATLAS-4153
> URL: https://issues.apache.org/jira/browse/ATLAS-4153
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Dharshana M Krishnamoorthy
> Priority: Major
>
> *Steps to re-produce:*
> * Stop kafka broker
> * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ...
> abc_table_n)
> * Make sure the data is spooled
> * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ...
> xyz_table_n)
> * Wait for 5 mins for the tables to reflect in atlas
> * Fire a basic search with abc_* prefix and verify the tables are created
> * Collect the createdTime of all the table and verify the order
> Here tables were created with prefix btwxb_table.
> Guid: a55ca87c-1122-4ed4-9812-5047cc8a7968 of *btwxb_table_6* has
> "createTime": *1612959902106*,
> {code:java}
> {
> "referredEntities": {...},
> "entity": {
> "typeName": "hive_table",
> "attributes": {
> "owner": "hrt_qa",
> "temporary": false,
> "lastAccessTime": 1612959527000,
> "aliases": null,
> "replicatedTo": null,
> "userDescription": null,
> "replicatedFrom": null,
> "qualifiedName": "default.btwxb_table_6@cm",
> "displayName": null,
> "columns": [
> {
> "guid": "3cf7b33b-7fb7-4de4-a967-7f89410b3a00",
> "typeName": "hive_column"
> },
> {
> "guid": "7c5b72ae-e100-467a-b3b2-caade0b7a7a1",
> "typeName": "hive_column"
> }
> ],
> "description": null,
> "viewExpandedText": null,
> "tableType": "MANAGED_TABLE",
> "sd": {
> "guid": "7a1e2b04-f786-4f42-a3c4-31a0ad872256",
> "typeName": "hive_storagedesc"
> },
> "createTime": 1612959527000,
> "name": "btwxb_table_6",
> "comment": null,
> "partitionKeys": [],
> "parameters": {...},
> "retention": 0,
> "viewOriginalText": null,
> "db": {
> "guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
> "typeName": "hive_db"
> }
> },
> "guid": "a55ca87c-1122-4ed4-9812-5047cc8a7968",
> "isIncomplete": false,
> "status": "ACTIVE",
> "createdBy": "hive",
> "updatedBy": "hrt_qa",
> "createTime": 1612959902106,
> "updateTime": 1612959911756,
> "version": 0,
> "relationshipAttributes": {...},
> "labels": []
> }
> } {code}
> Guid: 70738a7e-d29f-4234-82e7-adbdc4feaed6 of *btwxb_table_1* has
> "createTime": *1612959903892*,
> {code:java}
> {
> "referredEntities": {...},
> "entity": {
> "typeName": "hive_table",
> "attributes": {
> "owner": "hrt_qa",
> "temporary": false,
> "lastAccessTime": 1612959523000,
> "aliases": null,
> "replicatedTo": null,
> "userDescription": null,
> "replicatedFrom": null,
> "qualifiedName": "default.btwxb_table_1@cm",
> "displayName": null,
> "columns": [
> {
> "guid": "c283ee8c-d292-406a-a062-d81b12614636",
> "typeName": "hive_column"
> },
> {
> "guid": "cbb19940-e513-4295-a145-5b2d6691ea13",
> "typeName": "hive_column"
> }
> ],
> "description": null,
> "viewExpandedText": null,
> "tableType": "MANAGED_TABLE",
> "sd": {
> "guid": "69d7f6fc-8194-412f-a723-5b142ade1c31",
> "typeName": "hive_storagedesc"
> },
> "createTime": 1612959523000,
> "name": "btwxb_table_1",
> "comment": null,
> "partitionKeys": [],
> "parameters": {...},
> "retention": 0,
> "viewOriginalText": null,
> "db": {
> "guid": "a8a3227f-ee7a-49c2-8f9e-98534a3146eb",
> "typeName": "hive_db"
> }
> },
> "guid": "70738a7e-d29f-4234-82e7-adbdc4feaed6",
> "isIncomplete": false,
> "status": "ACTIVE",
> "createdBy": "hive",
> "updatedBy": "hrt_qa",
> "createTime": 1612959903892,
> "updateTime": 1612959919910,
> "version": 0,
> "relationshipAttributes": {...},
> "labels": []
> }
> } {code}
> Attributes's create time:
> *btwxb_table_1 : 1612959523000*
> *btwxb_table_6 : 1612959527000***
> In hive *btwxb_table_1* is created before *btwxb_table_6*
> Entity's create time:**
> *btwxb_table_1* has "createTime": *1612959903892*, and
> *btwxb_table_6* has "createTime": *1612959902106*, which is smaller than
> *btwxb_table_1's* createTime.
> *Which means table_6 is created before table_1 in atlas*
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
