See <https://builds.apache.org/job/carbondata-master-spark-2.2/org.apache.carbondata$carbondata-examples-spark2/1021/display/redirect?page=changes>
Changes: [QiangCai] CARBONDATA-2945] Support ingest JSON record using StreamSQL ------------------------------------------ [...truncated 215.73 KB...] | 6| Success|2018-09-18 23:05:...|2018-09-18 23:05:...| NA|COLUMNAR_V3| 2.27KB| 1.28KB| | 5|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 3.1|COLUMNAR_V3| 2.27KB| 1.28KB| | 4|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 3.1|COLUMNAR_V3| 2.27KB| 1.23KB| | 3.1|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 0.2|COLUMNAR_V3| 2.58KB| 1.24KB| | 3|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 3.1|COLUMNAR_V3| 2.27KB| 1.28KB| | 2|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 0.1|COLUMNAR_V3| 2.27KB| 1.28KB| | 1|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 0.1|COLUMNAR_V3| 2.27KB| 1.28KB| | 0.2| Success|2018-09-18 23:05:...|2018-09-18 23:05:...| NA|COLUMNAR_V3| 2.67KB| 1.29KB| | 0.1|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 0.2|COLUMNAR_V3| 2.58KB| 1.29KB| | 0|Compacted|2018-09-18 23:05:...|2018-09-18 23:05:...| 0.1|COLUMNAR_V3| 2.27KB| 1.28KB| +-----------------+---------+--------------------+--------------------+---------+-----------+---------+----------+ 18/09/18 23:05:22 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.tablecompaction_table 18/09/18 23:05:22 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.tablecompaction_table 18/09/18 23:05:22 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Compaction request received for table default.tablecompaction_table 18/09/18 23:05:22 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:22 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.tablecompaction_table 18/09/18 23:05:22 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.tablecompaction_table 18/09/18 23:05:22 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Compaction request received for table default.tablecompaction_table 18/09/18 23:05:22 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:22 AUDIT CarbonAlterTableCompactionCommand: [pietas.apache.org][jenkins][Thread-1]Compaction request received for table default.tablecompaction_table 18/09/18 23:05:23 AUDIT MergeBloomIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge bloom index 18/09/18 23:05:23 AUDIT CarbonTableCompactor: [pietas.apache.org][jenkins][Thread-1]Compaction request completed for table default.tablecompaction_table 18/09/18 23:05:23 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Merge index for compaction called 18/09/18 23:05:23 AUDIT CarbonStore$: [pietas.apache.org][jenkins][Thread-1]The clean files request has been received for default.tablecompaction_table 18/09/18 23:05:23 AUDIT CarbonStore$: [pietas.apache.org][jenkins][Thread-1]Clean files operation is success for default.tablecompaction_table. +-----------------+-------+--------------------+--------------------+---------+-----------+---------+----------+ |SegmentSequenceId| Status| Load Start Time| Load End Time|Merged To|File Format|Data Size|Index Size| +-----------------+-------+--------------------+--------------------+---------+-----------+---------+----------+ | 9|Success|2018-09-18 23:05:...|2018-09-18 23:05:...| NA|COLUMNAR_V3| 2.27KB| 1.28KB| | 8|Success|2018-09-18 23:05:...|2018-09-18 23:05:...| NA|COLUMNAR_V3| 2.27KB| 1.28KB| | 0.3|Success|2018-09-18 23:05:...|2018-09-18 23:05:...| NA|COLUMNAR_V3| 2.77KB| 1.29KB| +-----------------+-------+--------------------+--------------------+---------+-----------+---------+----------+ 18/09/18 23:05:23 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleting table [tablecompaction_table] under database [default] 18/09/18 23:05:23 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleted table [tablecompaction_table] under database [default] [32m- TableLevelCompactionOptionExample[0m 18/09/18 23:05:23 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [timeseriestable] 18/09/18 23:05:23 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [timeseriestable] 18/09/18 23:05:23 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [timeseriestable_agg0_hour] 18/09/18 23:05:23 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [timeseriestable_agg0_hour] 18/09/18 23:05:23 AUDIT CarbonCreateDataMapCommand: [pietas.apache.org][jenkins][Thread-1]DataMap agg0_hour successfully added 18/09/18 23:05:23 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [timeseriestable_agg0_day] 18/09/18 23:05:23 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [timeseriestable_agg0_day] 18/09/18 23:05:23 AUDIT CarbonCreateDataMapCommand: [pietas.apache.org][jenkins][Thread-1]DataMap agg0_day successfully added 18/09/18 23:05:23 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.timeseriestable 18/09/18 23:05:24 AUDIT CacheProvider: [pietas.apache.org][jenkins][Thread-1]The key carbon.input.segments.default.timeseriestable with value 0 added in the session param 18/09/18 23:05:24 AUDIT CacheProvider: [pietas.apache.org][jenkins][Thread-1]The key validate.carbon.input.segments.default.timeseriestable with value false added in the session param 18/09/18 23:05:24 AUDIT CacheProvider: [pietas.apache.org][jenkins][Thread-1]The key carbon.query.directQueryOnDataMap.enabled with value true added in the session param 18/09/18 23:05:24 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.timeseriestable_agg0_hour 18/09/18 23:05:25 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.timeseriestable_agg0_hour 18/09/18 23:05:25 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:25 AUDIT CacheProvider: [pietas.apache.org][jenkins][Thread-1]The key carbon.input.segments.default.timeseriestable_agg0_hour with value 0 added in the session param 18/09/18 23:05:25 AUDIT CacheProvider: [pietas.apache.org][jenkins][Thread-1]The key validate.carbon.input.segments.default.timeseriestable_agg0_hour with value false added in the session param 18/09/18 23:05:25 AUDIT CacheProvider: [pietas.apache.org][jenkins][Thread-1]The key carbon.query.directQueryOnDataMap.enabled with value true added in the session param 18/09/18 23:05:25 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.timeseriestable_agg0_day 18/09/18 23:05:26 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.timeseriestable_agg0_day 18/09/18 23:05:26 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:26 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.timeseriestable 18/09/18 23:05:26 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:26 AUDIT CacheProvider: [pietas.apache.org][jenkins][Thread-1]The key carbon.query.directQueryOnDataMap.enabled with value true added in the session param +--------+----------------------------+ |sum(age)|UDF:timeseries(mytime, hour)| +--------+----------------------------+ | 100| 2000-06-02 01:00:00| | 8| 2000-07-20 01:00:00| | 104| 2000-07-21 06:00:00| | 80| 2000-11-18 00:00:00| | 192| 2001-01-02 09:00:00| | 36| 2001-07-22 06:00:00| | 62| 2001-10-12 11:00:00| | 0| 2002-02-22 08:00:00| | 118| 2002-03-12 09:00:00| | 46| 2003-03-26 09:00:00| | 74| 2003-09-04 00:00:00| | 94| 2003-10-10 09:00:00| | 104| 2003-11-12 04:00:00| | 108| 2004-05-04 07:00:00| | 44| 2004-09-25 09:00:00| | 106| 2006-03-04 10:00:00| | 94| 2006-11-17 03:00:00| | 106| 2010-08-15 10:00:00| | 4| 2011-03-01 05:00:00| | 16| 2011-07-24 03:00:00| +--------+----------------------------+ only showing top 20 rows +------------------+----------------------------+ | avg(age)|UDF:timeseries(mytime, year)| +------------------+----------------------------+ |29.821739130434782| 2019-01-01 00:00:00| |30.871681415929203| 2006-01-01 00:00:00| | 30.140625| 2010-01-01 00:00:00| |29.534412955465587| 2018-01-01 00:00:00| |30.192156862745097| 2017-01-01 00:00:00| | 27.26007326007326| 2003-01-01 00:00:00| |30.232365145228215| 2005-01-01 00:00:00| |30.342629482071715| 2014-01-01 00:00:00| |30.090579710144926| 2016-01-01 00:00:00| |28.238970588235293| 2011-01-01 00:00:00| |30.192139737991265| 2015-01-01 00:00:00| |29.189075630252102| 2009-01-01 00:00:00| |29.184100418410043| 2002-01-01 00:00:00| | 30.50191570881226| 2004-01-01 00:00:00| | 29.064| 2000-01-01 00:00:00| |30.200803212851405| 2008-01-01 00:00:00| | 30.0| 2012-01-01 00:00:00| | 29.421875| 2007-01-01 00:00:00| | 29.05531914893617| 2013-01-01 00:00:00| |29.401673640167363| 2001-01-01 00:00:00| +------------------+----------------------------+ 18/09/18 23:05:27 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleting table [timeseriestable] under database [default] 18/09/18 23:05:27 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleting table [timeseriestable_agg0_hour] under database [default] 18/09/18 23:05:27 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleted table [timeseriestable_agg0_hour] under database [default] 18/09/18 23:05:27 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleting table [timeseriestable_agg0_day] under database [default] 18/09/18 23:05:27 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleted table [timeseriestable_agg0_day] under database [default] 18/09/18 23:05:27 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleted table [timeseriestable] under database [default] [32m- TimeSeriesPreAggregateTableExample[0m 18/09/18 23:05:27 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [persontable] 18/09/18 23:05:28 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [persontable] 18/09/18 23:05:28 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.persontable 18/09/18 23:05:29 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.persontable 18/09/18 23:05:29 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:29 AUDIT CarbonCreateDataMapCommand: [pietas.apache.org][jenkins][Thread-1]DataMap dm successfully added 18/09/18 23:05:30 AUDIT MergeBloomIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge bloom index +--------+ |count(1)| +--------+ | 10| +--------+ +--------+ |count(1)| +--------+ | 10| +--------+ time for query on table with lucene datamap table:0.224 time for query on table without lucene datamap table:0.33 +-----------------+-------------+ | id| name| +-----------------+-------------+ |which test1 good2|who and name2| |which test1 good2|who and name2| |which test1 good5|who and name0| |which test1 good5|who and name0| |which test1 good7|who and name5| |which test1 good7|who and name5| |which test1 good8|who and name7| |which test1 good8|who and name7| +-----------------+-------------+ +-----------------+-------------+ | id| name| +-----------------+-------------+ |which test1 good1|who and name3| |which test1 good1|who and name3| |which test1 good2|who and name2| |which test1 good2|who and name2| |which test1 good5|who and name0| |which test1 good5|who and name0| |which test1 good7|who and name5| |which test1 good7|who and name5| |which test1 good8|who and name7| |which test1 good8|who and name7| +-----------------+-------------+ 18/09/18 23:05:31 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleting table [persontable] under database [default] 18/09/18 23:05:31 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleted table [persontable] under database [default] [32m- LuceneDataMapExample[0m 18/09/18 23:05:31 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [origin_table] 18/09/18 23:05:31 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [origin_table] 18/09/18 23:05:31 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.origin_table 18/09/18 23:05:31 ERROR DataLoadExecutor: [Executor task launch worker for task 2380][partitionID:table;queryID:4997720772621469] Data Load is partially success for table origin_table 18/09/18 23:05:31 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.origin_table 18/09/18 23:05:31 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:31 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.origin_table 18/09/18 23:05:32 ERROR DataLoadExecutor: [Executor task launch worker for task 2382][partitionID:table;queryID:4997721042570180] Data Load is partially success for table origin_table 18/09/18 23:05:32 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.origin_table 18/09/18 23:05:32 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:32 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.origin_table 18/09/18 23:05:32 ERROR DataLoadExecutor: [Executor task launch worker for task 2384][partitionID:table;queryID:4997721372286029] Data Load is partially success for table origin_table 18/09/18 23:05:32 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.origin_table 18/09/18 23:05:32 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:32 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.origin_table 18/09/18 23:05:32 ERROR DataLoadExecutor: [Executor task launch worker for task 2386][partitionID:table;queryID:4997721632888125] Data Load is partially success for table origin_table 18/09/18 23:05:32 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.origin_table 18/09/18 23:05:32 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index +--------+ |count(1)| +--------+ | 40| +--------+ 18/09/18 23:05:32 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [external_table] 18/09/18 23:05:32 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [external_table] +--------+ |count(1)| +--------+ | 40| +--------+ 18/09/18 23:05:33 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.origin_table 18/09/18 23:05:33 ERROR DataLoadExecutor: [Executor task launch worker for task 2392][partitionID:table;queryID:4997722135511702] Data Load is partially success for table origin_table 18/09/18 23:05:33 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.origin_table 18/09/18 23:05:33 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:33 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.origin_table 18/09/18 23:05:33 ERROR DataLoadExecutor: [Executor task launch worker for task 2394][partitionID:table;queryID:4997722442487970] Data Load is partially success for table origin_table 18/09/18 23:05:33 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.origin_table 18/09/18 23:05:33 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index +--------+ |count(1)| +--------+ | 60| +--------+ 18/09/18 23:05:33 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleting table [origin_table] under database [default] 18/09/18 23:05:33 AUDIT CarbonDropTableCommand: [pietas.apache.org][jenkins][Thread-1]Deleted table [origin_table] under database [default] [32m- ExternalTableExample[0m 18/09/18 23:05:33 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [maintable] 18/09/18 23:05:33 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [maintable] 18/09/18 23:05:33 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Creating Table with Database name [default] and Table name [dimtable] 18/09/18 23:05:33 AUDIT CarbonCreateTableCommand: [pietas.apache.org][jenkins][Thread-1]Table created with Database name [default] and Table name [dimtable] 18/09/18 23:05:33 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.maintable 18/09/18 23:05:34 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.maintable 18/09/18 23:05:34 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index 18/09/18 23:05:34 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load request has been received for table default.dimtable 18/09/18 23:05:35 AUDIT CarbonDataRDDFactory$: [pietas.apache.org][jenkins][Thread-1]Data load is successful for default.dimtable 18/09/18 23:05:35 AUDIT MergeIndexEventListener: [pietas.apache.org][jenkins][Thread-1]Load post status event-listener called for merge index [31m- MVDataMapExample *** FAILED ***[0m [31m java.lang.ClassNotFoundException: org.apache.carbondata.mv.datamap.MVDataMapProvider[0m [31m at java.net.URLClassLoader.findClass(URLClassLoader.java:381)[0m [31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)[0m [31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)[0m [31m at java.lang.Class.forName0(Native Method)[0m [31m at java.lang.Class.forName(Class.java:348)[0m [31m at org.apache.spark.util.Utils$.classForName(Utils.scala:230)[0m [31m at org.apache.spark.util.CarbonReflectionUtils$.createObject(CarbonReflectionUtils.scala:305)[0m [31m at org.apache.carbondata.spark.util.CarbonScalaUtil$.createDataMapProvider(CarbonScalaUtil.scala:480)[0m [31m at org.apache.carbondata.spark.util.CarbonScalaUtil.createDataMapProvider(CarbonScalaUtil.scala)[0m [31m at org.apache.carbondata.datamap.DataMapManager.getDataMapProvider(DataMapManager.java:57)[0m [31m ...[0m [36mRun completed in 1 minute, 58 seconds.[0m [36mTotal number of tests run: 18[0m [36mSuites: completed 2, aborted 0[0m [36mTests: succeeded 17, failed 1, canceled 0, ignored 0, pending 0[0m [31m*** 1 TEST FAILED ***[0m [JENKINS] Recording test results
