imay commented on issue #3549:
URL:
https://github.com/apache/incubator-doris/issues/3549#issuecomment-626520097
@morningman
Wonderful proposal.
I have some questions about this.
> Write the result to the local file system of a BE. And provide users with
an http connection for downloading.
If the result file will be downloaded through HTTP, what's the
authentication information will be attached for the download request. If there
is an authentication check for the download request, the relation between user
and result file will be saved. Maybe through rule, such as all result belong to
one user locates in the same directory. Or FE will store all the authentication
information.
Besides that, it is not a clever idea to store huge data in one single local
file. So we should have a solution to support split a huge result into many
files in local filesystem. And if it is supported, what user gets is not a
single URL, he will get a list of URLs.
And there is another choice that we don't support unload to local file, we
only can unload file to remote storage, such as S3, HDFS.
> {"download_url" : "http://be:http_port/file.."}
1. I think download operation should be routed by FE. And we can't give user
the absolute path in the URL. It is a good that FE can rewrite the path.
2. Usually unload is a heavy operation, if user lose the returned download
URL, what left for him/her is to do this unload again, which is not efficient.
If we make sure to support unload to local file, what I suggest is that we
should support a command for users to list all unload result information.
3. As far as I know, it is hard to get information from OK message of MySQL
protocol. It is not a good design to return load-id for `insert into select`
command. So I think we should avoid this in unload operation.
> `result_size_in_bytes`
I think "max_result_bytes" is better.
> `result_file_expiration_time`
`result_expiration_second`. Better to give unit information in name.
> `file name`
fix typo to `name_prefix` and I think there is no need to add TTL in file
name.
After all, I think it is a good way to support remote FS first, because it
will avoid us to do many things unrelated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
