Re: Camel AWS-S3 prefixes do not accept wildcards
There is no method for sync from CLI in SDK. I'll add a note in the prefix description about wildcards and regexes missing support from SDK. Thanks for the suggestion! Il giorno mer 13 mar 2024 alle ore 21:13 Andrea Cosentino ha scritto: > I'll investigate if there is a way of using sync somewhat through the sdk. > I'll keep you posted > > Il mer 13 mar 2024, 20:47 Mitch Trachtenberg ha > scritto: > >> My mistake. What I'd thought was just aws cli ls was actually output >> piped >> through a grep. >> >> Again, though, for many people used to non-AWS file listings this may come >> as a surprise, so it would IMO be helpful to tell people up front that >> wildcards, regexes, etc... won't work. And thanks for letting me know >> that >> the asterisk is a legal character in an AWS S3 object name. >> >> My situation is I expect only objects whose names match a particular >> pattern, but I want to prevent the download of any surprise objects that >> might get deposited. It looks like list, eliminate, and download the ones >> you want is going to have to be the approach to accomplish that. >> >> On Wed, Mar 13, 2024 at 12:13 PM Andrea Cosentino >> wrote: >> >> > Hi, >> > In the CLI it was escalated but it doesn't seem to be supported: >> > https://github.com/aws/aws-cli/issues/3784 >> > >> > Usually the CLI/SDK/Rest are aligned, so if it was possible natively >> > through CLI it would have been possible through SDK. >> > >> > The prefix is only a String in v1 and v2 and most importantly * is valid >> > character in the key name for an S3 object. >> > >> > My suggestion is to list the objects by prefix and then check through >> > regular expression if the key name responds to your regex or not. >> > >> > As far as I know and see in the documentation, list objects through >> > wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but >> > sync is a different command than list objects and I guess is a different >> > Java library too. >> > >> > Cheers >> > >> > Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < >> > mjt...@gmail.com> ha scritto: >> > >> > > Thanks, Claus. I'm sure you're correct and I've looked at the >> > > documentation for ListObjectsRequest. >> > > >> > > >> > >> https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html >> > > >> > > But it seems like something that is such a common need that it would >> be >> > > helpful IMO to include a note to that effect in the docs. >> > > >> > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen >> > > wrote: >> > > >> > > > Hi >> > > > >> > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its >> > > > depends on if this SDK has any kind of support for wildcards. >> > > > As you write its likely it does not have that. >> > > > >> > > > You can try to dive into the AWS SDK and see more deeper. >> > > > >> > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg < >> mjt...@gmail.com> >> > > > wrote: >> > > > >> > > > > I cannot specifically find this stated in either AWS S3 or Camel >> > > > > documentation, but it appears that I can give a prefix 1234- to >> > > restrict >> > > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot >> give a >> > > > prefix >> > > > > - to restrict Camel's downloads to files beginning with four >> > > > characters >> > > > > and a hyphen. >> > > > > >> > > > > I believe AWS CLI does provide a way to do this. If I'm correct, >> it >> > > > would >> > > > > be helpful if this could be added to the documentation. >> > > > > >> > > > > If someone were to want to instruct me in where/how to do that, >> I'd >> > be >> > > > > happy to generate a pull request, but it might be easier to just >> add: >> > > > > "Note: due to AWS S3 limitations, the prefix parameter does not >> work >> > > with >> > > > > wildcards." >> > > > > >> > > > > It would be even more helpful, of course, if the component were >> > > modified >> > > > in >> > > > > some way to allow for wildcards. >> > > > > >> > > > >> > > > >> > > > -- >> > > > Claus Ibsen >> > > > - >> > > > @davsclaus >> > > > Camel in Action 2: https://www.manning.com/ibsen2 >> > > > >> > > >> > >> >
Re: Camel AWS-S3 prefixes do not accept wildcards
I'll investigate if there is a way of using sync somewhat through the sdk. I'll keep you posted Il mer 13 mar 2024, 20:47 Mitch Trachtenberg ha scritto: > My mistake. What I'd thought was just aws cli ls was actually output piped > through a grep. > > Again, though, for many people used to non-AWS file listings this may come > as a surprise, so it would IMO be helpful to tell people up front that > wildcards, regexes, etc... won't work. And thanks for letting me know that > the asterisk is a legal character in an AWS S3 object name. > > My situation is I expect only objects whose names match a particular > pattern, but I want to prevent the download of any surprise objects that > might get deposited. It looks like list, eliminate, and download the ones > you want is going to have to be the approach to accomplish that. > > On Wed, Mar 13, 2024 at 12:13 PM Andrea Cosentino > wrote: > > > Hi, > > In the CLI it was escalated but it doesn't seem to be supported: > > https://github.com/aws/aws-cli/issues/3784 > > > > Usually the CLI/SDK/Rest are aligned, so if it was possible natively > > through CLI it would have been possible through SDK. > > > > The prefix is only a String in v1 and v2 and most importantly * is valid > > character in the key name for an S3 object. > > > > My suggestion is to list the objects by prefix and then check through > > regular expression if the key name responds to your regex or not. > > > > As far as I know and see in the documentation, list objects through > > wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but > > sync is a different command than list objects and I guess is a different > > Java library too. > > > > Cheers > > > > Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < > > mjt...@gmail.com> ha scritto: > > > > > Thanks, Claus. I'm sure you're correct and I've looked at the > > > documentation for ListObjectsRequest. > > > > > > > > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html > > > > > > But it seems like something that is such a common need that it would be > > > helpful IMO to include a note to that effect in the docs. > > > > > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen > > > wrote: > > > > > > > Hi > > > > > > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > > > > depends on if this SDK has any kind of support for wildcards. > > > > As you write its likely it does not have that. > > > > > > > > You can try to dive into the AWS SDK and see more deeper. > > > > > > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > > > > > wrote: > > > > > > > > > I cannot specifically find this stated in either AWS S3 or Camel > > > > > documentation, but it appears that I can give a prefix 1234- to > > > restrict > > > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give > a > > > > prefix > > > > > - to restrict Camel's downloads to files beginning with four > > > > characters > > > > > and a hyphen. > > > > > > > > > > I believe AWS CLI does provide a way to do this. If I'm correct, > it > > > > would > > > > > be helpful if this could be added to the documentation. > > > > > > > > > > If someone were to want to instruct me in where/how to do that, I'd > > be > > > > > happy to generate a pull request, but it might be easier to just > add: > > > > > "Note: due to AWS S3 limitations, the prefix parameter does not > work > > > with > > > > > wildcards." > > > > > > > > > > It would be even more helpful, of course, if the component were > > > modified > > > > in > > > > > some way to allow for wildcards. > > > > > > > > > > > > > > > > > -- > > > > Claus Ibsen > > > > - > > > > @davsclaus > > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > > > > > > >
Re: Camel AWS-S3 prefixes do not accept wildcards
My mistake. What I'd thought was just aws cli ls was actually output piped through a grep. Again, though, for many people used to non-AWS file listings this may come as a surprise, so it would IMO be helpful to tell people up front that wildcards, regexes, etc... won't work. And thanks for letting me know that the asterisk is a legal character in an AWS S3 object name. My situation is I expect only objects whose names match a particular pattern, but I want to prevent the download of any surprise objects that might get deposited. It looks like list, eliminate, and download the ones you want is going to have to be the approach to accomplish that. On Wed, Mar 13, 2024 at 12:13 PM Andrea Cosentino wrote: > Hi, > In the CLI it was escalated but it doesn't seem to be supported: > https://github.com/aws/aws-cli/issues/3784 > > Usually the CLI/SDK/Rest are aligned, so if it was possible natively > through CLI it would have been possible through SDK. > > The prefix is only a String in v1 and v2 and most importantly * is valid > character in the key name for an S3 object. > > My suggestion is to list the objects by prefix and then check through > regular expression if the key name responds to your regex or not. > > As far as I know and see in the documentation, list objects through > wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but > sync is a different command than list objects and I guess is a different > Java library too. > > Cheers > > Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < > mjt...@gmail.com> ha scritto: > > > Thanks, Claus. I'm sure you're correct and I've looked at the > > documentation for ListObjectsRequest. > > > > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html > > > > But it seems like something that is such a common need that it would be > > helpful IMO to include a note to that effect in the docs. > > > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen > > wrote: > > > > > Hi > > > > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > > > depends on if this SDK has any kind of support for wildcards. > > > As you write its likely it does not have that. > > > > > > You can try to dive into the AWS SDK and see more deeper. > > > > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > > > wrote: > > > > > > > I cannot specifically find this stated in either AWS S3 or Camel > > > > documentation, but it appears that I can give a prefix 1234- to > > restrict > > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a > > > prefix > > > > - to restrict Camel's downloads to files beginning with four > > > characters > > > > and a hyphen. > > > > > > > > I believe AWS CLI does provide a way to do this. If I'm correct, it > > > would > > > > be helpful if this could be added to the documentation. > > > > > > > > If someone were to want to instruct me in where/how to do that, I'd > be > > > > happy to generate a pull request, but it might be easier to just add: > > > > "Note: due to AWS S3 limitations, the prefix parameter does not work > > with > > > > wildcards." > > > > > > > > It would be even more helpful, of course, if the component were > > modified > > > in > > > > some way to allow for wildcards. > > > > > > > > > > > > > -- > > > Claus Ibsen > > > - > > > @davsclaus > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > > >
Re: Camel AWS-S3 prefixes do not accept wildcards
Hi, In the CLI it was escalated but it doesn't seem to be supported: https://github.com/aws/aws-cli/issues/3784 Usually the CLI/SDK/Rest are aligned, so if it was possible natively through CLI it would have been possible through SDK. The prefix is only a String in v1 and v2 and most importantly * is valid character in the key name for an S3 object. My suggestion is to list the objects by prefix and then check through regular expression if the key name responds to your regex or not. As far as I know and see in the documentation, list objects through wildcards is not supported with AWS S3 ls, it works with AWS S3 sync but sync is a different command than list objects and I guess is a different Java library too. Cheers Il giorno mer 13 mar 2024 alle ore 19:37 Mitch Trachtenberg < mjt...@gmail.com> ha scritto: > Thanks, Claus. I'm sure you're correct and I've looked at the > documentation for ListObjectsRequest. > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html > > But it seems like something that is such a common need that it would be > helpful IMO to include a note to that effect in the docs. > > On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen > wrote: > > > Hi > > > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > > depends on if this SDK has any kind of support for wildcards. > > As you write its likely it does not have that. > > > > You can try to dive into the AWS SDK and see more deeper. > > > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > > wrote: > > > > > I cannot specifically find this stated in either AWS S3 or Camel > > > documentation, but it appears that I can give a prefix 1234- to > restrict > > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a > > prefix > > > - to restrict Camel's downloads to files beginning with four > > characters > > > and a hyphen. > > > > > > I believe AWS CLI does provide a way to do this. If I'm correct, it > > would > > > be helpful if this could be added to the documentation. > > > > > > If someone were to want to instruct me in where/how to do that, I'd be > > > happy to generate a pull request, but it might be easier to just add: > > > "Note: due to AWS S3 limitations, the prefix parameter does not work > with > > > wildcards." > > > > > > It would be even more helpful, of course, if the component were > modified > > in > > > some way to allow for wildcards. > > > > > > > > > -- > > Claus Ibsen > > - > > @davsclaus > > Camel in Action 2: https://www.manning.com/ibsen2 > > >
Re: Camel AWS-S3 prefixes do not accept wildcards
Thanks, Claus. I'm sure you're correct and I've looked at the documentation for ListObjectsRequest. https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ListObjectsRequest.html But it seems like something that is such a common need that it would be helpful IMO to include a note to that effect in the docs. On Wed, Mar 13, 2024 at 11:30 AM Claus Ibsen wrote: > Hi > > Camel uses the AWS Java SDK and it set the prefix on this SDK so its > depends on if this SDK has any kind of support for wildcards. > As you write its likely it does not have that. > > You can try to dive into the AWS SDK and see more deeper. > > On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg > wrote: > > > I cannot specifically find this stated in either AWS S3 or Camel > > documentation, but it appears that I can give a prefix 1234- to restrict > > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a > prefix > > - to restrict Camel's downloads to files beginning with four > characters > > and a hyphen. > > > > I believe AWS CLI does provide a way to do this. If I'm correct, it > would > > be helpful if this could be added to the documentation. > > > > If someone were to want to instruct me in where/how to do that, I'd be > > happy to generate a pull request, but it might be easier to just add: > > "Note: due to AWS S3 limitations, the prefix parameter does not work with > > wildcards." > > > > It would be even more helpful, of course, if the component were modified > in > > some way to allow for wildcards. > > > > > -- > Claus Ibsen > - > @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >
Re: Camel AWS-S3 prefixes do not accept wildcards
Hi Camel uses the AWS Java SDK and it set the prefix on this SDK so its depends on if this SDK has any kind of support for wildcards. As you write its likely it does not have that. You can try to dive into the AWS SDK and see more deeper. On Wed, Mar 13, 2024 at 7:19 PM Mitch Trachtenberg wrote: > I cannot specifically find this stated in either AWS S3 or Camel > documentation, but it appears that I can give a prefix 1234- to restrict > Camel's (4.3.0) downloads to files starting 1234-, but cannot give a prefix > - to restrict Camel's downloads to files beginning with four characters > and a hyphen. > > I believe AWS CLI does provide a way to do this. If I'm correct, it would > be helpful if this could be added to the documentation. > > If someone were to want to instruct me in where/how to do that, I'd be > happy to generate a pull request, but it might be easier to just add: > "Note: due to AWS S3 limitations, the prefix parameter does not work with > wildcards." > > It would be even more helpful, of course, if the component were modified in > some way to allow for wildcards. > -- Claus Ibsen - @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Camel AWS-S3 prefixes do not accept wildcards
I cannot specifically find this stated in either AWS S3 or Camel documentation, but it appears that I can give a prefix 1234- to restrict Camel's (4.3.0) downloads to files starting 1234-, but cannot give a prefix - to restrict Camel's downloads to files beginning with four characters and a hyphen. I believe AWS CLI does provide a way to do this. If I'm correct, it would be helpful if this could be added to the documentation. If someone were to want to instruct me in where/how to do that, I'd be happy to generate a pull request, but it might be easier to just add: "Note: due to AWS S3 limitations, the prefix parameter does not work with wildcards." It would be even more helpful, of course, if the component were modified in some way to allow for wildcards.
