-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43575/
-----------------------------------------------------------
(Updated Feb. 25, 2016, 12:07 p.m.)
Review request for Ambari, DIPAYAN BHOWMICK, Pallav Kulshreshtha, Srimanth
Gunturi, and Yusaku Sako.
Changes
-------
removed conflict from ResultsPaginationController. Accepted "their" changes.
Bugs: AMBARI-15046
https://issues.apache.org/jira/browse/AMBARI-15046
Repository: ambari
Description
-------
Support added to the file upload feature for the following:
upload table format now supports JSON/XML/CSV format
File source can be local upload or HDFS
Internal Hive storage type of ORC and others should be supported, with ORC as
default.
Following is the workflow
1. user selects upload source Local / HDFS
2. user selects the input file format CSV/JSON/XML
3. if local then he uploads the file else enters the HDFS file path.
4. the file is partially read from browser or HDFS.
5. preview is generated using Parsers
6. user enters database,tablename,table column names,column types, precision
and scale and hive storage type as per requirement.
7. user clicks on Upload table button.
8. actual table (user selected hive storage format) and temporary table (Stored
as CSV) are created.
9. the file is again read from browser or HDFS stream and JSON/XML streams are
converted to CSV streams.
In case of HDFS upload + CSV format + isFirstRowHeader == false, hive's LOAD
DATA INPATH statement is used instead of re-reading the stream.
10. this stream is uploaded into the temporary table.
11. insert from temporary table to actual table is executed.
12. on success the temporary table is deleted and workflow completes.
Added in this patch:
Parsing :
org/apache/ambari/view/hive/resources/uploads/parsers : contains parser for
CSV/JSON and XML
apache commons CSV parser is used for CSV.
gson JsonReader is used for JSON
and XMLEventReader of javax for XML stream parsing. The registered parser will
automatically be used.
All parsers ultimately create Iterators for iterating through the rows of the
table.
Data in Iterator is converted into CSV stream using TableDataReader so as to
upload it into HDFS
QueryGenerator.java has simple methods for generating various hive queries.
UploadService.java provides the complete REST interface. whole workflow is UI
driven and the server is stateless.
JobService.java : one more REST call is added for finding the status of job
"{jobId}/status".
UI:
controller/upload-table.js is the main file which contains most of the code for
workflow
upload-table.hbs contains the template.
adapter/upload-table.js contains network call specific code
input-header and radio-button components are added.
Diffs (updated)
-----
contrib/views/hive/pom.xml 9f3f1f7
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/client/Row.java
35f216b
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/jobs/JobService.java
fd69893
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/CSVParser.java
388cf53
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/ColumnDescriptionImpl.java
50f5036
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/DataParser.java
5f2db55
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/HiveFileType.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/IParser.java
c478b70
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/ParseOptions.java
2ec3b1b
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/ParseUtils.java
aea370e
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/QueryGenerator.java
98616cf
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/TableDataReader.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/TableInfo.java
ed4943d
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/TableInput.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/UploadFromHdfsInput.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/UploadService.java
8b5b851
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/DataParser.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/EndOfDocumentException.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/IParser.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/ParseOptions.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/ParseUtils.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/Parser.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/PreviewData.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/RowIterator.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/RowMapIterator.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/csv/CSVIterator.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/csv/CSVParser.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/json/JSONIterator.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/json/JSONParser.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/xml/XMLIterator.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/xml/XMLParser.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/DeleteQueryInput.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/InsertFromQueryInput.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/LoadQueryInput.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/QueryGenerator.java
PRE-CREATION
contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/TableInfo.java
PRE-CREATION
contrib/views/hive/src/main/resources/ui/hive-web/app/adapters/upload-table.js
6a9c54b
contrib/views/hive/src/main/resources/ui/hive-web/app/components/input-header.js
PRE-CREATION
contrib/views/hive/src/main/resources/ui/hive-web/app/components/radio-button.js
PRE-CREATION
contrib/views/hive/src/main/resources/ui/hive-web/app/controllers/upload-table.js
ab7b934
contrib/views/hive/src/main/resources/ui/hive-web/app/initializers/i18n.js
bd0e6e6
contrib/views/hive/src/main/resources/ui/hive-web/app/services/job.js 3a1f82a
contrib/views/hive/src/main/resources/ui/hive-web/app/styles/app.scss 31dafdd
contrib/views/hive/src/main/resources/ui/hive-web/app/templates/components/input-header.hbs
PRE-CREATION
contrib/views/hive/src/main/resources/ui/hive-web/app/templates/upload-table.hbs
d811e34
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/CSVParserTest.java
PRE-CREATION
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserCSVTest.java
PRE-CREATION
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserJSONTest.java
PRE-CREATION
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserTest.java
3d77d29
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserXMLTest.java
PRE-CREATION
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/JsonParserTest.java
PRE-CREATION
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/QueryGeneratorTest.java
PRE-CREATION
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/TableDataReaderTest.java
PRE-CREATION
contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/XMLParserTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/43575/diff/
Testing
-------
Server side Unit test cases added for testing Parsers and TableDataReader.
CSVParserTest.java
DataParserCSVTest.java
DataParserJSONTest.java
DataParserTest.java
DataParserXMLTest.java
JsonParserTest.java
QueryGeneratorTest.java
TableDataReaderTest.java
XMLParserTest.java
Thanks,
Nitiraj Rathore
