mmmmxa opened a new issue, #4105: URL: https://github.com/apache/iggy/issues/4105
### Description This is still not final and a WIP. As mentioned a number of time on Discord, for the last few weeks I was working on trying to make Delta sink run on a real EC2 instance writing to S3. The implementation turned out to have a substantial amount of issues that render the sink not practical to use in production: 1. S3 storage type mandates access key and secret key, use of which is highly discouraged by AWS. The modern practice is to assign a role to an instance that is going to perform the write, for which you should simply write into the destination bucket without any credentials. 2. The existence of the destination delta table is not checked, which causes obscure errors when the table doesn't exist in the destination S3 URL. The table is not automatically created either, but this is likely to be a future feature. 3. When set in production, `aws_s3_endpoint_url` is confusing S3 API to understand the region setting which doesn't allow the code to open the table for writing. This setting should be documented better (and studied as there seem to be a couple of other quirks) as removing it usually fixes the problem. 4. If a connectors runtime instantiated with a config containing topics that do not exist yet, and then tries to connect to the iggy-server, it stalls indefinitely and the sink is not writing anymore, which seems to be an offset issue, but the cause of this is yet to be studied in more detail. 5. Current implementation allows to have a setup when no fields from the JSONs inside of the topic and the fields (columns) of the table schema overlap, causing the system to write a table full of nulls. This needs to be studied further. 6. The defaults for `batch_length` and `poll_interval` have to be adjusted for Delta tables. Writing the data with the standard `batch_length == 100` and `poll_interval == 5ms` is going to create lots and lots of small files in Delta, which is causing very slow reading on the side of Delta readers. The default of `batch_length == 10000` and `poll_interval == 3s` is much more adequate for most use cases. All of these will likely be bundled in a number of different PRs but the grouping is still TBD. ### Affected area / component Connectors ### Proposed solution This is still not final and a WIP. Items are corresponding to the enumeration in the issue description. 1. Stop mandating the S3 access and secret keys. Mention the two access options in the docs. Suggest the sufficient policy configuration to be given to the instance. 2. Check the table existence after opening, write a test for it. 3. Study and document the usage of `aws_s3_endpoint_url` configuration setting. 4. The stalling after failure to find a stream / topic by the connectors runtime has to be studied and most likely there needs to be an error with a retry period or something like this. 5. Study the schema coercion process further to see the exact nature of the problem. I think it would make sense to implement an error on no fields overlapping as this results in all-null records made into the delta table. 6. Change defaults in the documentation. ### Alternatives considered _No response_ ### Contribution - [x] I'm willing to submit a pull request to implement this feature ### Good first issue - [ ] I think this could be a good first issue for a new contributor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
