thanks a lot, Hao, finally solved this problem, changes of CSVSerDe are here:
https://github.com/chutium/csv-serde/commit/22c667c003e705613c202355a8791978d790591e
btw, add jar in spark hive or hive-thriftserver always doesn't work, we
build the spark with libraryDependencies += csv-serde ...
or
Hi Cheng, thank you very much for helping me to finally find out the secret
of this magic...
actually we defined this external table with
SID STRING
REQUEST_ID STRING
TIMES_DQ TIMESTAMP
TOTAL_PRICE FLOAT
...
using desc table ext_fullorders it is only shown as
[# col_name
different dataTypes,
feature or a bug? really strange and surprised...
Hi Cheng, thank you very much for helping me to finally find out the secret of
this magic...
actually we defined this external table with
SID STRING
REQUEST_ID STRING
TIMES_DQ TIMESTAMP
TOTAL_PRICE FLOAT
I believe in your case, the “magic” happens in TableReader.fillObject
https://github.com/apache/spark/blob/4fa2fda88fc7beebb579ba808e400113b512533b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L706-L712.
Here we unwrap the field value according to the object inspector of that
is there any dataType auto convert or detect or something in HiveContext ?all
columns of a table is defined as string in hive metastoreone column is
total_price with values like 123.45, then this column will be recognized as
dataType Float in HiveContext...this is a feature or a bug? it really
is there any dataType auto convert or detect or something in HiveContext ?
all columns of a table is defined as string in hive metastore
one column is total_price with values like 123.45, then this column will be
recognized as dataType Float in HiveContext...
this is a feature or a bug? it
oops, i tried on a managed table, column types will not be changed
so it is mostly due to the serde lib CSVSerDe
(https://github.com/ogrodnek/csv-serde/blob/master/src/main/java/com/bizo/hive/serde/csv/CSVSerde.java#L123)
or maybe CSVReader from opencsv?...
but if the columns are defined as
