xuchuanyin created CARBONDATA-1700:
--------------------------------------
Summary: Failed to load data to existed table after spark session
restarted
Key: CARBONDATA-1700
URL: https://issues.apache.org/jira/browse/CARBONDATA-1700
Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 1.3.0
Reporter: xuchuanyin
Assignee: xuchuanyin
Fix For: 1.3.0
# scenario
I encounterd loading data to existed carbondata table failure after query the
table after restarting spark session. I have this failure in spark local mode
(found it during local test) and haven't test in other scenarioes.
The problem can be reproduced by following steps：
0. START: start a session;
1. CREATE: create table `t1`;
2. LOAD: create a dataframe and write apppend to `t1`;
3. STOP: stop current session;
4. START: start a session;
5. QUERY: query table `t1`; ---- This step is essential to reproduce the
problem.
6. LOAD: create a dataframe and write append to `t1`; --- This step will be
failed.
Error will be thrown in Step6. The error message in console looks like
```
java.lang.NullPointerException was thrown.
java.lang.NullPointerException
at
org.apache.spark.sql.execution.command.management.LoadTableCommand.processData(LoadTableCommand.scala:92)
at
org.apache.spark.sql.execution.command.management.LoadTableCommand.run(LoadTableCommand.scala:60)
at
org.apache.spark.sql.CarbonDataFrameWriter.loadDataFrame(CarbonDataFrameWriter.scala:141)
at
org.apache.spark.sql.CarbonDataFrameWriter.writeToCarbonFile(CarbonDataFrameWriter.scala:50)
at
org.apache.spark.sql.CarbonDataFrameWriter.appendToCarbonFile(CarbonDataFrameWriter.scala:42)
at org.apache.spark.sql.CarbonSource.createRelation(CarbonSource.scala:110)
at
org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:426)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)
```
The following code can be pasted in `TestLoadDataFrame.scala` to reproduce this
problem —— but keep
in mind you should manually run the first test and then the second in different
iteration (to make sure that the sparksession is restarted).
```
test("prepare") {
sql("drop table if exists carbon_stand_alone")
sql( "create table if not exists carbon_stand_alone (c1 string, c2 string,
c3 int)" +
" stored by 'carbondata'").collect()
sql("select * from carbon_stand_alone").show()
df.write
.format("carbondata")
.option("tableName", "carbon_stand_alone")
.option("tempCSV", "false")
.mode(SaveMode.Append)
.save()
}
test("test load dataframe after query") {
sql("select * from carbon_stand_alone").show()
// the following line will cause failure
df.write
.format("carbondata")
.option("tableName", "carbon_stand_alone")
.option("tempCSV", "false")
.mode(SaveMode.Append)
.save()
// if it works fine, it sould be true
checkAnswer(
sql("select count(*) from carbon_stand_alone where c3 > 500"), Row(31500
* 2)
)
}
```
# ANALYSE
I went through the code and found the problem was caused by NULL
`tableProperties` in `tablemeta: tableMeta.carbonTable.getTableInfo
.getFactTable.getTableProperties` (we will name it `propertyInTableInfo`
for short) is null in Line89 in `LoadTableCommand.scala`.
After debug, I found that the `propertyInTableInfo` sett in
`CarbonTableInputFormat.setTableInfo(...)` had the correct value. But
`CarbonTableInputFormat.getTableInfo(...)` had the incorrect value. The setter
is used to serialized TableInfo, while the getter is used to deserialized
TableInfo ———— That means there are something wrong in
serialization-deserialization.
Keep diving into the code, I found that serialization and deserialization in
`TableSchema`, a member of `TableInfo`, ignores the `tableProperties` member,
thus causing this value empty after deserialization. Since this value has not
been initialized in construtor, so the value remains `NULL` and cause the NPE
problem.
# RESOLVE
1. Initialize `tableProperties` in `TableSchema`
2. Include `tableProperties` in serialization-deserialization of `TableSchema`
# Notes
Although the bug has been fix, I still can't understand why the problem can be
triggered in above way.
Tests need the sparksession to be restarted, which is impossible currently, so
no tests will be added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
