[
https://issues.apache.org/jira/browse/ATLAS-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307446#comment-15307446
]
Hemanth Yamijala commented on ATLAS-760:
----------------------------------------
Actually, I dug into the patch a little more. From what [~ayubkhan] reported,
these properties are coming as part of the structure called *table parameters*
in Hive. [This
link|https://books.google.co.in/books?id=n8FffirFaQIC&pg=PA54&lpg=PA54&dq=hive+last_modified_by&source=bl&ots=JJTEGaVVqD&sig=pz4_ouLepwhhGSod2s8B_yTSOLM&hl=en&sa=X&ved=0ahUKEwjAj7_I8oPNAhWIt48KHXx9DdoQ6AEILjAD#v=onepage&q=hive%20last_modified_by&f=false]
shows the said two parameters are automatically added by Hive. So, it appears
they should be available in the table parameters map of a Hive table? However,
in {{HiveMetaStoreBridge.createOrUpdateTableInstance}}, we do copy the
parameters as is:
{code}
tableReference.set(HiveDataModelGenerator.PARAMETERS,
hiveTable.getParameters());
{code}
And while these seem important enough to call-out separately, I think we should
check whether we are getting these in the parameters already and are missing
them due to some other issue? If we can confirm we're not getting them in the
above method, then we could go ahead with this approach.
Another issue I see is that these parameters are not captured when we are
importing tables from hive (as opposed to HiveHook).
> Missing table parameters(last_modified_by and last_modified_time)
> -----------------------------------------------------------------
>
> Key: ATLAS-760
> URL: https://issues.apache.org/jira/browse/ATLAS-760
> Project: Atlas
> Issue Type: Bug
> Affects Versions: 0.7-incubating
> Reporter: Ayub Khan
> Assignee: Vimal Sharma
> Priority: Critical
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-760.patch
>
>
> Missing table parameters(last_modified_by and last_modified_time)
> Hive table metadata showing vboth params..
> {noformat}
> 0: jdbc:hive2://localhost:10000/default> describe formatted table3_s1vxgkt0bh;
> OK
> +-------------------------------+--------------------------------------------------------------+--------------------------------+--+
> | col_name | data_type
> | comment |
> +-------------------------------+--------------------------------------------------------------+--------------------------------+--+
> | # col_name | data_type
> | comment |
> | | NULL
> | NULL |
> | viewtime | int
> | |
> | userid | bigint
> | |
> | page_url | string
> | |
> | referrer_url | string
> | |
> | ip | string
> | |
> | | NULL
> | NULL |
> | # Partition Information | NULL
> | NULL |
> | # col_name | data_type
> | comment |
> | | NULL
> | NULL |
> | dt | string
> | |
> | country | string
> | partitioned columns comments. |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | Owner: | apathan
> | NULL |
> | CreateTime: | Mon May 09 14:42:34 IST 2016
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Protect Mode: | None
> | NULL |
> | Retention: | 0
> | NULL |
> | Location: |
> hdfs://localhost:9000/user/hive/warehouse/table3_s1vxgkt0bh | NULL
> |
> | Table Type: | MANAGED_TABLE
> | NULL |
> | Table Parameters: | NULL
> | NULL |
> | | last_modified_by
> | apathan |
> | | last_modified_time
> | 1462785155 |
> | | transient_lastDdlTime
> | 1462785155 |
> | | NULL
> | NULL |
> | # Storage Information | NULL
> | NULL |
> | SerDe Library: |
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
> |
> | InputFormat: | org.apache.hadoop.mapred.TextInputFormat
> | NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
> |
> | Compressed: | No
> | NULL |
> | Num Buckets: | -1
> | NULL |
> | Bucket Columns: | []
> | NULL |
> | Sort Columns: | []
> | NULL |
> | Storage Desc Params: | NULL
> | NULL |
> | | serialization.format
> | 1 |
> +-------------------------------+--------------------------------------------------------------+--------------------------------+--+
> 38 rows selected (0.101 seconds)
> {noformat}
> Entity api response showing missing parameters
> {noformat}
> curl
> http://localhost:21000/api/atlas/entities/b17b2c93-dbf7-4592-b59e-4614c6e3d37e
> | python -m json.tool
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left Speed
> 100 6508 0 6508 0 0 4152 0 --:--:-- 0:00:01 --:--:-- 4150
> {
> "GUID": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "definition": {
> "id": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_table",
> "values": {
> "columns": [
> {
> "id": {
> "id": "f118ccf4-bd8b-4a0e-af35-17a5c5018e06",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "viewtime",
> "qualifiedName":
> "default.table3_s1vxgkt0bh.viewtime@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "int"
> }
> },
> {
> "id": {
> "id": "4b0d47e1-16ba-4cdd-85a5-6f35a968c0a3",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "userid",
> "qualifiedName":
> "default.table3_s1vxgkt0bh.userid@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "bigint"
> }
> },
> {
> "id": {
> "id": "ca652a37-a5b8-4563-be27-4c6adb20f2d9",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "page_url",
> "qualifiedName":
> "default.table3_s1vxgkt0bh.page_url@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "string"
> }
> },
> {
> "id": {
> "id": "050d14cb-6434-4230-896e-063c6df04de9",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "referrer_url",
> "qualifiedName":
> "default.table3_s1vxgkt0bh.referrer_url@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "string"
> }
> },
> {
> "id": {
> "id": "39d7eede-e6e1-4a82-ab4a-0dbbe211cd42",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "ip",
> "qualifiedName":
> "default.table3_s1vxgkt0bh.ip@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "string"
> }
> }
> ],
> "comment": null,
> "createTime": "2016-05-09T09:12:34.000Z",
> "db": {
> "id": "802c10fe-10b3-423e-a587-19c3d45a3ca5",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_db",
> "version": 0
> },
> "description": null,
> "lastAccessTime": "2016-05-09T09:12:34.000Z",
> "name": "default.table3_s1vxgkt0bh@primary",
> "owner": "apathan",
> "parameters": {
> "transient_lastDdlTime": "1462785154"
> },
> "partitionKeys": [
> {
> "id": {
> "id": "ba216f52-6ea0-4b00-831f-87edd9b3d6ab",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "dt",
> "qualifiedName":
> "default.table3_s1vxgkt0bh.dt@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "string"
> }
> },
> {
> "id": {
> "id": "fe1bcce2-5ad9-4506-9cd5-2db1e61d4de5",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": "partitioned columns comments.",
> "name": "country",
> "qualifiedName":
> "default.table3_s1vxgkt0bh.country@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "string"
> }
> }
> ],
> "retention": 0,
> "sd": {
> "id": {
> "id": "5499b90c-570f-4911-abe4-cca0cc749db3",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_storagedesc",
> "version": 0
> },
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_storagedesc",
> "values": {
> "bucketCols": null,
> "compressed": false,
> "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
> "location":
> "hdfs://localhost:9000/user/hive/warehouse/db2s1vxgkt0bh.db/table2_s1vxgkt0bh",
> "numBuckets": -1,
> "outputFormat":
> "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
> "parameters": null,
> "qualifiedName":
> "default.table3_s1vxgkt0bh@primary_storage",
> "serdeInfo": {
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
> "typeName": "hive_serde",
> "values": {
> "name": null,
> "parameters": {
> "serialization.format": "1"
> },
> "serializationLib":
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
> }
> },
> "sortCols": null,
> "storedAsSubDirectories": false,
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass":
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> }
> }
> },
> "tableName": "table3_s1vxgkt0bh",
> "tableType": "MANAGED_TABLE",
> "temporary": false,
> "viewExpandedText": null,
> "viewOriginalText": null
> }
> },
> "requestId": "qtp1307096070-2007 - afa719f8-ad88-4293-a4d6-16c6a72190ed"
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
