hi,all:
when I call api df.write.parquet ,there is alot of small files : how can
I merge then into on file ? I tried df.coalesce(1).write.parquet ,but it will
get error some times
Container exited with a non-zero exit code 143
more an more...
-rw-r--r-- 2 hadoop supergroup 14.5 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00165-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 16.4 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00166-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 17.1 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00167-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 14.2 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00168-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 15.7 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00169-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 14.4 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00170-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 17.1 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00171-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 15.7 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00172-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 16.0 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00173-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 17.1 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00174-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 14.0 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00175-0f61afe4-23e8-40bb-b30b-09652ca677bc.snappy.parquet
-rw-r--r-- 2 hadoop supergroup 15.7 K 2016-11-10 15:11
/parquetdata/weixin/biztags/biztag2/part-r-00176-0f61afe4-23e8-40bb-b30b-09652ca677bc
more an more...
2016-11-10
lk_spark
