rubenssoto edited a comment on issue #2508:
URL: https://github.com/apache/hudi/issues/2508#issuecomment-770475266
I made more tests and I only had the problem when my bulk insert operation
has the option hoodie.datasource.write.row.writer.enable true.
I only had this problem on this table, this table has a column type array
with that option false the column on hive:
`order_details_line_items`
array<struct<product_variant_id:int,item_inventory_id:int,price:decimal(14,2),total_price:decimal(14,2),subtotal_price:decimal(14,2),total_item_price:decimal(14,2),total_tax:decimal(14,2),total_coupon_discount:decimal(14,2),total_offer_discount:decimal(14,2),total_discount:decimal(14,2),quantity:int,description:string,short_description:string,image_url:string,title:string,subtitle:string,product:struct<product_id:int,title:string,tags:string,label:string,image_url:string,description:string,short_description:string,rgb:boolean,has_fixed_price:boolean>,category:struct<id:int,title:string>,brand:struct<id:int,title:string>,applicable_discount:struct<discount_value:decimal(14,2),discount_type:string,discount_value_type:int,presented_discount_value:decimal(14,2),final_price:decimal(14,2),final_unit_price:decimal(14,2)>>>
with that option true:
`order_details_line_items`
array<struct<product_variant_id:int,item_inventory_id:int,price:decimal(14,2),total_price:decimal(14,2),subtotal_price:decimal(14,2),total_item_price:decimal(14,2),total_tax:decimal(14,2),total_coupon_discount:decimal(14,2),total_offer_discount:decimal(14,2),total_discount:decimal(14,2),quantity:int,description:string,short_description:string,image_url:string,title:string,subtitle:string,product:struct<product_id:int,title:string,tags:string,label:string,image_url:string,description:string,short_description:string,rgb:boolean,has_fixed_price:boolean>,category:struct<id:int,title:string>,brand:struct<id:int,title:string>,applicable_discount:struct<discount_value:decimal(14,2),discount_type:string,discount_value_type:int,presented_discount_value:decimal(14,2),final_price:decimal(14,2),final_unit_price:decimal(14,2)>>>
The original column it is a json and I struct the column in spark with this
schema:
StructField(
"line_items",
ArrayType(
StructType(
List(
StructField("product_variant_id", IntegerType),
StructField("item_inventory_id", IntegerType),
StructField("price", DecimalType(14, 2)),
StructField("total_price", DecimalType(14, 2)),
StructField("subtotal_price", DecimalType(14, 2)),
StructField("total_item_price", DecimalType(14, 2)),
StructField("total_tax", DecimalType(14, 2)),
StructField("total_coupon_discount", DecimalType(14, 2)),
StructField("total_offer_discount", DecimalType(14, 2)),
StructField("total_discount", DecimalType(14, 2)),
StructField("quantity", IntegerType),
StructField("description", StringType),
StructField("short_description", StringType),
StructField("image_url", StringType),
StructField("title", StringType),
StructField("subtitle", StringType),
StructField(
"product",
StructType(
List(
StructField("product_id", IntegerType),
StructField("title", StringType),
StructField("tags", StringType),
StructField("label", StringType),
StructField("image_url", StringType),
StructField("description", StringType),
StructField("short_description", StringType),
StructField("rgb", BooleanType),
StructField("has_fixed_price", BooleanType)
)
)
)
I will try to convert the column to string....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
