thomasrebele commented on code in PR #6825: URL: https://github.com/apache/hive/pull/6825#discussion_r4143050421
########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/postgresql.conf: ########## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# general + +listen_addresses = '*' +dynamic_shared_memory_type = posix + +# performance + +work_mem = 256MB +shared_buffers = 128MB + +# reduce WAL and archiving + +checkpoint_timeout=1d +max_wal_size = 64MB +min_wal_size = 32MB +wal_level = minimal +max_wal_senders = 0 +wal_buffers = -1 +archive_mode = off +archive_command = '/bin/true' Review Comment: I added those to prevent archiving from being turned on in the future. ########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/postgresql.conf: ########## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# general + +listen_addresses = '*' +dynamic_shared_memory_type = posix + +# performance + +work_mem = 256MB +shared_buffers = 128MB + +# reduce WAL and archiving + +checkpoint_timeout=1d +max_wal_size = 64MB +min_wal_size = 32MB +wal_level = minimal +max_wal_senders = 0 +wal_buffers = -1 Review Comment: Yes, I fear that's a side-effect of the configuration. I had specifically set the wal size quite low to avoid using disk space unnecessarily. This is especially helpful if a developer keeps a container of the image around to do some experiments manually. ########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/postgresql.conf: ########## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# general + +listen_addresses = '*' +dynamic_shared_memory_type = posix + +# performance + +work_mem = 256MB +shared_buffers = 128MB + +# reduce WAL and archiving + +checkpoint_timeout=1d +max_wal_size = 64MB +min_wal_size = 32MB +wal_level = minimal +max_wal_senders = 0 +wal_buffers = -1 +archive_mode = off +archive_command = '/bin/true' + +# locale settings + +log_timezone = 'Etc/UTC' +datestyle = 'iso, mdy' +timezone = 'Etc/UTC' +lc_messages = 'en_US.utf8' +lc_monetary = 'en_US.utf8' +lc_numeric = 'en_US.utf8' +lc_time = 'en_US.utf8' +default_text_search_config = 'pg_catalog.english' Review Comment: I had encountered an `PSQLException: FATAL: invalid value for parameter "TimeZone": "US/Pacific"` The postgres18 image does not provide the timezone US/Pacific. The US/Pacific timezone might come from the [pom.xml file](https://github.com/apache/hive/blob/289d7280cb02afc251de506d65a79a5307b4f229/pom.xml#L1902). I didn't investigate why it is passed to the Postgres container. As I switched to another postgresql.conf file, I copied the values from the config file in the container (see this [gist](https://gist.github.com/thomasrebele/489dcb50637d5e63cfbda3f70f648eb7)). Some of them redundantly specify the default value. I had deleted some of the redundancy, but not all of them. ########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/postgresql.conf: ########## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# general + +listen_addresses = '*' +dynamic_shared_memory_type = posix + +# performance + +work_mem = 256MB +shared_buffers = 128MB Review Comment: It is also specified in the configuration file that is created by the default postgres:18-alpine image during initialization. If necessary, I can remove all settings that just set the default value. ########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/postgresql.conf: ########## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# general + +listen_addresses = '*' +dynamic_shared_memory_type = posix Review Comment: See https://github.com/apache/hive/pull/6825/changes#r4143127930. ########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/entrypoint.sh: ########## @@ -13,10 +15,9 @@ # See the License for the specific language governing permissions and # limitations under the License. -#!/bin/bash +if [ -f /tmp/metastore_db.zstd ]; then + zstdcat /tmp/metastore_db.zstd | tar -C /var/lib/postgresql/ -x + rm /tmp/metastore_db.zstd +fi Review Comment: Yes, this is due to the commands. The first is just a warning. I'm not sure about the purpose of the second. The archive is extracted and Postgres can read the data without issues, so I would just accept these two. If these warnings are too bothersome, I can look into how to avoid them. I specifically used a custom entry point to be able to extract an zstd compressed archive that contains the Postgres database. Zstd compresses the 1.2G database to 543M. I did a small experiment (just the compression tool itself): * gzip compresses the database to 642M, and takes 5.15s to extract * zstd compresses the database to 543M, and takes 1s to extract Afaik, Docker still uses gzip, so using zstd here saves 100M. ########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/postgresql.conf: ########## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# general + +listen_addresses = '*' +dynamic_shared_memory_type = posix + +# performance + +work_mem = 256MB Review Comment: If that's too much, we could reduce it to 128MB or 64MB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
