Do you have the logs of the containers? This seems like a Memory issue. 2016-11-10 7:28 GMT+00:00 lk_spark <lk_sp...@163.com>:
> hi,all: > when I call api df.write.parquet ,there is alot of small files : how > can I merge then into on file ? I tried df.coalesce(1).write.parquet ,but > it will get error some times > > Container exited with a non-zero exit code 143 > > more an more... > -rw-r--r-- 2 hadoop supergroup 14.5 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00165-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 16.4 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00166-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 17.1 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00167-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 14.2 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00168-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 15.7 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00169-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 14.4 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00170-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 17.1 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00171-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 15.7 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00172-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 16.0 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00173-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 17.1 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00174-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 14.0 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00175-0f61afe4- > 23e8-40bb-b30b-09652ca677bc.snappy.parquet > -rw-r--r-- 2 hadoop supergroup 15.7 K 2016-11-10 15:11 > /parquetdata/weixin/biztags/biztag2/part-r-00176-0f61afe4- > 23e8-40bb-b30b-09652ca677bc > more an more... > 2016-11-10 > ------------------------------ > lk_spark >
