----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43575/#review120158 -----------------------------------------------------------
Ship it! Ship It! - Srimanth Gunturi On Feb. 15, 2016, 12:24 p.m., Nitiraj Rathore wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43575/ > ----------------------------------------------------------- > > (Updated Feb. 15, 2016, 12:24 p.m.) > > > Review request for Ambari, DIPAYAN BHOWMICK, Pallav Kulshreshtha, Srimanth > Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15046 > https://issues.apache.org/jira/browse/AMBARI-15046 > > > Repository: ambari > > > Description > ------- > > Support added to the file upload feature for the following: > upload table format now supports JSON/XML/CSV format > File source can be local upload or HDFS > Internal Hive storage type of ORC and others should be supported, with ORC as > default. > > Following is the workflow > 1. user selects upload source Local / HDFS > 2. user selects the input file format CSV/JSON/XML > 3. if local then he uploads the file else enters the HDFS file path. > 4. the file is partially read from browser or HDFS. > 5. preview is generated using Parsers > 6. user enters database,tablename,table column names,column types, precision > and scale and hive storage type as per requirement. > 7. user clicks on Upload table button. > 8. actual table (user selected hive storage format) and temporary table > (Stored as CSV) are created. > 9. the file is again read from browser or HDFS stream and JSON/XML streams > are converted to CSV streams. > In case of HDFS upload + CSV format + isFirstRowHeader == false, hive's > LOAD DATA INPATH statement is used instead of re-reading the stream. > 10. this stream is uploaded into the temporary table. > 11. insert from temporary table to actual table is executed. > 12. on success the temporary table is deleted and workflow completes. > > Added in this patch: > Parsing : > org/apache/ambari/view/hive/resources/uploads/parsers : contains parser for > CSV/JSON and XML > apache commons CSV parser is used for CSV. > gson JsonReader is used for JSON > and XMLEventReader of javax for XML stream parsing. The registered parser > will automatically be used. > All parsers ultimately create Iterators for iterating through the rows of the > table. > > Data in Iterator is converted into CSV stream using TableDataReader so as to > upload it into HDFS > > QueryGenerator.java has simple methods for generating various hive queries. > > UploadService.java provides the complete REST interface. whole workflow is UI > driven and the server is stateless. > > JobService.java : one more REST call is added for finding the status of job > "{jobId}/status". > > UI: > controller/upload-table.js is the main file which contains most of the code > for workflow > upload-table.hbs contains the template. > adapter/upload-table.js contains network call specific code > input-header and radio-button components are added. > > > Diffs > ----- > > contrib/views/hive/pom.xml 9f3f1f7 > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/client/Row.java > 35f216b > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/jobs/JobService.java > fd69893 > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/jobs/ResultsPaginationController.java > 84dec4d > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/CSVParser.java > 388cf53 > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/ColumnDescriptionImpl.java > 50f5036 > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/DataParser.java > 5f2db55 > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/HiveFileType.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/IParser.java > c478b70 > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/ParseOptions.java > 2ec3b1b > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/ParseUtils.java > aea370e > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/QueryGenerator.java > 98616cf > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/TableDataReader.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/TableInfo.java > ed4943d > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/TableInput.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/UploadFromHdfsInput.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/UploadService.java > 8b5b851 > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/DataParser.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/EndOfDocumentException.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/IParser.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/ParseOptions.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/ParseUtils.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/Parser.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/PreviewData.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/RowIterator.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/RowMapIterator.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/csv/CSVIterator.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/csv/CSVParser.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/json/JSONIterator.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/json/JSONParser.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/xml/XMLIterator.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/parsers/xml/XMLParser.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/DeleteQueryInput.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/InsertFromQueryInput.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/LoadQueryInput.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/QueryGenerator.java > PRE-CREATION > > contrib/views/hive/src/main/java/org/apache/ambari/view/hive/resources/uploads/query/TableInfo.java > PRE-CREATION > > contrib/views/hive/src/main/resources/ui/hive-web/app/adapters/upload-table.js > 6a9c54b > > contrib/views/hive/src/main/resources/ui/hive-web/app/components/input-header.js > PRE-CREATION > > contrib/views/hive/src/main/resources/ui/hive-web/app/components/radio-button.js > PRE-CREATION > > contrib/views/hive/src/main/resources/ui/hive-web/app/controllers/upload-table.js > ab7b934 > contrib/views/hive/src/main/resources/ui/hive-web/app/initializers/i18n.js > bd0e6e6 > contrib/views/hive/src/main/resources/ui/hive-web/app/services/job.js > 3a1f82a > contrib/views/hive/src/main/resources/ui/hive-web/app/styles/app.scss > 31dafdd > > contrib/views/hive/src/main/resources/ui/hive-web/app/templates/components/input-header.hbs > PRE-CREATION > > contrib/views/hive/src/main/resources/ui/hive-web/app/templates/upload-table.hbs > d811e34 > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/CSVParserTest.java > PRE-CREATION > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserCSVTest.java > PRE-CREATION > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserJSONTest.java > PRE-CREATION > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserTest.java > 3d77d29 > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/DataParserXMLTest.java > PRE-CREATION > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/JsonParserTest.java > PRE-CREATION > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/QueryGeneratorTest.java > PRE-CREATION > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/TableDataReaderTest.java > PRE-CREATION > > contrib/views/hive/src/test/java/org/apache/ambari/view/hive/resources/upload/XMLParserTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/43575/diff/ > > > Testing > ------- > > Server side Unit test cases added for testing Parsers and TableDataReader. > > CSVParserTest.java > DataParserCSVTest.java > DataParserJSONTest.java > DataParserTest.java > DataParserXMLTest.java > JsonParserTest.java > QueryGeneratorTest.java > TableDataReaderTest.java > XMLParserTest.java > > > Thanks, > > Nitiraj Rathore > >
