Hi,
I think apache/amoro is a good name, ams and optimizer are just internal
components of amoro, similar to Flink JM/TM, so they can be referred to by
amoro, and the image does have all its features.
Also, for QuickStart, I recommend changing its filesystem to minio or using an
official HDFS image, which should not be maintained by Amoro.
Best regards,
BaiyangTX
---- Replied Message ----
| From | Qishang Zhong<zhongqish...@gmail.com> |
| Date | 4/30/2024 10:39 |
| To | <dev@amoro.apache.org> |
| Subject | Re: Adjust docker images of apache amoro project |
Hi,
IMO, I agree with jinsong's point of view. Using FlinkCDC to ingest data
produces too many Delete Files, which is the vast majority of reasons why
users choose amoro.
Therefore, it would be better to provide a quick start image and provide
the most realistic scene demo to users who are exposed to amoro for the
first time.
Regarding mirroring, I agree with Baiyang's point of view. Only keep
production-available mirrors under apache, and for quick start we can put
them under amoro (including datanode and namenode)
- apache/amoro-ams (Just `amoro` may not be very clear, because amoro also
includes many things like mixed format, optimizer)
- apache/amoro-optimizer-flink (maybe need to rename)
- apache/amoro-optimizer-spark (maybe need to rename)
- amoro/amoro-quick-demo
- amoro/amoro-datanode
- amoro/amoro-datanode
Best regards,
Qishang Zhong
Jinsong Zhou <jinsongz...@apache.org> 于2024年4月30日周二 10:01写道：
Hi,
Executing Insert/Delete/Update SQL through the Terminal can also trigger
self-optimizing and greatly reduce the difficulty of the quick demo
process.
However, CDC streaming ingestion remains a scenario that most users are
very interested in, and I'm not sure whether it should be removed from the
quick demo.
I'd like to hear more inputs from other developers and users.
Best,
Jinsong
On Mon, Apr 29, 2024 at 3:07 PM Gang Huang <tcodehu...@gmail.com> wrote:
Thanks for your rely.
I agree with the vast majority of opinions. But I think we should
simplify
the quick start(whether installation or usage) for a newbie. From this
point, we can remove amoro-quick-demo image instead of using amoro-ams
image.
Users can insert/delete/update rows by ams' Terminal page, then we can
trigger minor/major processes.
.
Jinsong Zhou <jinsongz...@apache.org> 于2024年4月29日周一 14:44写道：
Hi,
Thanks a lot for driving this.
I agree that we should keep the amoro-ams(renamed from amoro),
amoro-flink-optimizer(renamed from flink-optimizer), and
amoro-spark-optimizer(renamed from spark-optimizer) images.
Besides, considering that the quick demo now requires additional usage
of
the Flink engine to complete CDC data ingestion, we still need the
amoro-quick-demo image until we plan to adjust the quick demo process.
Best,
Jinsong
On Sat, Apr 27, 2024 at 1:30 PM BaiyangTX <xiangneb...@163.com> wrote:
Hi,
The docker image currently maintained in the project does have room
for
further optimization.
amoro: This is the core image of the project. It provides AMS
deployment
in early versions. After this PR will be merged (
https://github.com/apache/amoro/pull/2695), it can also directly
provide
the deployment of Optimizer on K8S. , this should also be the
recommended
using in the future. I recommend maintaining this image in the apache
repo
in the future, using apache/amoro as the image name. Here you can
refer
to
kyuubi (https://hub.docker.com/r/apache/kyuubi)
optimizer-flink/optimizer-spark: Provides images of optimizer
deployment
under different computing engines. I also recommend that these two
images
be maintained under the apache repo, using
apache/amoro-flink-optimizer
and
apache/amoro-spark-optimizer as the image id.
quickstart: Used to demonstrate the QuickStart part of the official
website. This image is based on the amoro image and includes
computing
engines such as flink/spark and connectors such as iceberg. It is
recommended to use amoro's repo maintenance for this part and use
amoro/quickstart as the image name.
namenode/datanode: These two images are used to provide an HDFS
environment in the qucikstart demonstration. I suggest modifying the
current quickstart process, using minio as the quickstart
environment,
and
no longer maintaining these two images.
The above are my personal suggestions.
Kind Regards,
baiyangtx
---- Replied Message ----
| From | Gang Huang<tcodehu...@gmail.com> |
| Date | 4/26/2024 17:03 |
| To | <dev@amoro.apache.org> |
| Subject | Adjust docker images of apache amoro project |
Hi,
Currently, there are up to 6 docker images in apache amoro project.
But
in
my opinion, only amoro, optimizer-flink and optimizer-spark maybe are
needed. Furthermore, we have to change the images' final
names(amoro-ams,
amoro-optimizer-flink, amoro-optimizer-spark) to identify them when
uploading them into docker hub.
Thus, we can simplify the quick start process for better user
experience,
just like iceberg/risingwave quickstart:
https://iceberg.apache.org/spark-quickstart/
https://docs.risingwave.com/docs/current/get-started/
Please feel free to contribute your suggestions.
Kind Regards,
Gang Huang
--
Best Regards,
Qishang Zhong
