i changed to
ArrayType(ArrayType(ArrayType(IntegerType()))) , still get same error
Thank you for responding
On Thursday, July 27, 2023 at 06:58:09 PM GMT+8, Adrian Pop-Tifrea
<poptifreaadr...@gmail.com> wrote:
Hello,
when you said your pandas Dataframe has 10 rows, does that mean it contains 10
images? Because if that's the case, then you'd want ro only use 3 layers of
ArrayType when you define the schema.
Best regards,Adrian
On Thu, Jul 27, 2023, 11:04 second_co...@yahoo.com.INVALID
<second_co...@yahoo.com.invalid> wrote:
i have panda dataframe with column 'image' using numpy.ndarray. shape is (500,
333, 3) per image. my panda dataframe has 10 rows, thus, shape is (10, 500,
333, 3)
when using spark.createDataframe(panda_dataframe, schema), i need to specify
the schema,
schema = StructType([
StructField("image",
ArrayType(ArrayType(ArrayType(ArrayType(IntegerType())))), nullable=False)
])
i get error
raise TypeError(
, TypeError: field image:
ArrayType(ArrayType(ArrayType(ArrayType(IntegerType(), True), True), True),
True) can not accept object array([[[14, 14, 14],...
Can advise how to set schema for image with numpy.ndarray ?
