[GitHub] [druid] wangxiaobaidu11 edited a comment on pull request #12159: Add Spark Writer support.

Wed, 16 Feb 2022 23:57:58 -0800 


wangxiaobaidu11 edited a comment on pull request #12159:
URL: https://github.com/apache/druid/pull/12159#issuecomment-1042657139


   @JulianJaffePinterest   I tested it this way
   `Dataset<Row> dataset = sparkSession.createDataFrame(list, schema);`
   ` Dataset<Row> bucketedDataset =  
dataset.withColumn("quick_col",SparkUdfs.bucketRow().apply(functions.col("_timeBucketCol"),functions.lit("millis"),functions.lit("DAY")));`
   `    Dataset<Row> partitionedDataSet1 = 
bucketedDataset.repartition(functions.col("quick_col")).drop(functions.col("quick_col"));`
   `    Dataset<Row> partitionedDataSet2 = 
partitionedDataSet1.repartition(10);`  // Increase parallelism
   `    
partitionedDataSet2.write().format("druid").mode(SaveMode.Overwrite).options(map).save();`
   but in log I find it ：
   
![image](https://user-images.githubusercontent.com/24448732/154428611-e60f7ac0-9ed8-413e-9cc6-3e21002755ce.png)
   
![image](https://user-images.githubusercontent.com/24448732/154429248-9fab0f17-d6f3-4f49-8d21-87c5857154e5.png)
   I think it should be five segments, but only two segment
   Corresponding code block：
   
![image](https://user-images.githubusercontent.com/24448732/154428964-b4c38aee-da5f-4866-aa11-69fb5b5b2d6d.png)
   Code url: 
   
https://github.com/JulianJaffePinterest/druid/blob/2dd1cda37289b780328e26d08bedfb1f57bac0e3/spark/src/main/scala/org/apache/druid/spark/utils/SegmentRationalizer.scala
   What configuration items do I need to modify ？
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to