john-bodley commented on a change in pull request #4488: utf8 fixes to csv ->
hive upload
URL:
https://github.com/apache/incubator-superset/pull/4488#discussion_r278787852
##########
File path: superset/db_engine_specs.py
##########
@@ -873,11 +873,12 @@ def get_column_names(filepath):
s3 = boto3.client('s3')
location = os.path.join('s3a://', bucket_path, upload_prefix,
table_name)
s3.upload_file(
- upload_path, 'airbnb-superset',
+ upload_path, bucket_path,
os.path.join(upload_prefix, table_name, filename))
sql = """CREATE EXTERNAL TABLE {table_name} ( {schema_definition} )
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS
- TEXTFILE LOCATION '{location}'""".format(**locals())
+ TEXTFILE LOCATION '{location}'
+ tblproperties ('skip.header.line.count'='1')""".format(**locals())
Review comment:
Note the `skip.header.line.count` property will only work in Presto for
v0.199 or later per
[here](https://prestodb.github.io/docs/current/release/release-0.199.html#hive-changes).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
