Radhika Kundam created ATLAS-4293:
-------------------------------------
Summary: Table changes are not propagated to Atlas in Create Table
with External Storage Handler and few scenarios of Alter table
Key: ATLAS-4293
URL: https://issues.apache.org/jira/browse/ATLAS-4293
Project: Atlas
Issue Type: Bug
Components: atlas-core
Affects Versions: 2.1.0
Reporter: Radhika Kundam
Assignee: Radhika Kundam
*Scenario-1:* Create Hive Table with external storage handler
In this case there is no entity created for the external hbase path, though
there is atlas entity for hbase table.Also no lineage present on atlas for this
table to external path.
Query for Hive Table with external storage handler:
*create table default.tbl29_hbs(key int, value string)*
*STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'*
*WITH SERDEPROPERTIES (*
*"hbase.columns.mapping" = ":key,cf:name",*
*"hbase.table.name" = "default:hbs_tbl29"*
*);*
Below are the Alter table scenarios which are not propagated to Atlas
*Scenario-2:* SET SERDE
CREATE TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 (a int, b int, c int,
d int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
ALTER TABLE db2k9eooxraj5.column_serde_changek9eooxraj5 SET SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 61 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected
[org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe] but found
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
*Scenario-3:* SET FILEFORMAT TEXTFILE
CREATE TABLE db2k9eooxraj5.text_table_k9eooxraj5 (a int, b int, c int, d int, e
int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
alter table db2k9eooxraj5.text_table_k9eooxraj5 SET FILEFORMAT TEXTFILE
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.788 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected
[org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat] but found
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat
mismatch expected [org.apache.hadoop.mapred.TextInputFormat] but found
[org.apache.hadoop.mapred.SequenceFileInputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
*Scenario-4:* SET FILEFORMAT SEQUENCEFILE
CREATE TABLE db2k9eooxraj5.sequence_table_k9eooxraj5 (a int, b int, c int, d
int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
alter table db2k9eooxraj5.sequence_table_k9eooxraj5 SET FILEFORMAT SEQUENCEFILE
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.796 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
outputFormat mismatch expected
[org.apache.hadoop.mapred.SequenceFileOutputFormat] but found
[org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
*Scenario-5:* SET FILEFORMAT PARQUET
CREATE TABLE db2k9eooxraj5.parquet_table_k9eooxraj5 (a int, b int, c int, d
int, e int, f int) ROW FORMAT
SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' stored as SequenceFile
alter table db2k9eooxraj5.parquet_table_k9eooxraj5 SET FILEFORMAT PARQUET
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 60.803 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
serdeInfo serializationLib mismatch expected
[org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe] but found
[org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe], outputFormat mismatch
expected [org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat] but
found [org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat], inputFormat
mismatch expected
[org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat] but found
[org.apache.hadoop.mapred.SequenceFileInputFormat]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
*Scenario-6:* ALTER TABLE using CLUSTERED BY
create table db2k9eooxraj5.transaction_table_k9eooxraj5 (EmployeeID
Int,FirstName String,Designation String, Salary Int,Department String)
clustered by (department) into 3 buckets stored as orc TBLPROPERTIES
('transactional'='true')
ALTER TABLE db2k9eooxraj5.cluster_by_k9eooxraj5 CLUSTERED BY (viewTime, userid)
SORTED BY (page_url) INTO 48 BUCKETS
*Assert Error:*
alterTableTest(org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest)
Time elapsed: 61.25 sec <<< FAILURE!
java.lang.AssertionError: The following asserts failed:
bucketCols mismatch: lists don't have the same size expected [2] but found [0],
numBuckets mismatch expected [48] but found [-1]
at org.testng.asserts.SoftAssert.assertAll(SoftAssert.java:38)
at
org.apache.atlas.regression.tests.HiveIntegrationAlterSupportTest.alterTableTest(HiveIntegrationAlterSupportTest.java:382)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
