jackye1995 commented on a change in pull request #3175:
URL: https://github.com/apache/iceberg/pull/3175#discussion_r717274097
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -328,6 +330,15 @@ private static InputStream uncheckedInputStream(File file)
{
}
}
+ private void createStagingDirectoryIfNotExists() throws SecurityException {
+ if (!stagingDirectory.exists()) {
+ boolean createdStagingDirectory = stagingDirectory.mkdirs();
+ if (createdStagingDirectory) {
+ LOG.info("Successfully created staging directory: {}",
stagingDirectory.getAbsolutePath());
+ }
Review comment:
else, add a LOG saying "Staging directory {} creation failed, or it is
created by another process"
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -328,6 +330,15 @@ private static InputStream uncheckedInputStream(File file)
{
}
}
+ private void createStagingDirectoryIfNotExists() throws SecurityException {
+ if (!stagingDirectory.exists()) {
Review comment:
add a LOG saying something like "staging directoy {} not exist, trying
to create one"
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -170,11 +171,12 @@ public void write(byte[] b, int off, int len) throws
IOException {
}
}
- private void newStream() throws IOException {
+ private void newStream() throws IOException, SecurityException {
Review comment:
Sorry for the back and forth, I read the documentation for mkdirs, it
seems like SecurityException only catches JVM level permission and it might
still just return false for OS level permission failure, so it's hard to have a
consistent behavior for error handling. Because of that, plus the fact that
SecurityException is a runtime exception, I think we can remove the special
handling of it and just let it throw to the top level.
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputFile.java
##########
@@ -62,6 +63,9 @@ public PositionOutputStream createOrOverwrite() {
return new S3OutputStream(client(), uri(), awsProperties());
} catch (IOException e) {
throw new UncheckedIOException("Failed to create output stream for
location: " + uri(), e);
+ } catch (SecurityException e) {
Review comment:
as discussed in the block below, we can remove this catch.
AccessControlException also hides the original exception, so better to not use
it.
##########
File path: api/src/main/java/org/apache/iceberg/io/OutputFile.java
##########
@@ -48,6 +49,7 @@
*
* @return an output stream that can report its position
* @throws RuntimeIOException If the implementation throws an {@link
IOException}
+ * @throws AccessControlException If the implementation throws an {@link
SecurityException}
Review comment:
as discussed in the block below, we can just document @throws
SecurityException if staging directory creation fails due to missing JVM level
permission.
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -328,6 +330,15 @@ private static InputStream uncheckedInputStream(File file)
{
}
}
+ private void createStagingDirectoryIfNotExists() throws SecurityException {
+ if (!stagingDirectory.exists()) {
+ boolean createdStagingDirectory = stagingDirectory.mkdirs();
+ if (createdStagingDirectory) {
+ LOG.info("Successfully created staging directory: {}",
stagingDirectory.getAbsolutePath());
+ }
Review comment:
actually, we can do better than just logging. When creation fails, we
can check directory existence again, if it exists then it's created by another
process, otherwise it's still an issue, and we can throw an IOException with
error message indicating staging directory creation fails for some unknown
reasons.
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -328,6 +330,26 @@ private static InputStream uncheckedInputStream(File file)
{
}
}
+ private void createStagingDirectoryIfNotExists() throws IOException,
SecurityException {
+ if (!stagingDirectory.exists()) {
+ LOG.info("Staging directory: {} does not exist, trying to create one",
+ stagingDirectory.getAbsolutePath());
+ boolean createdStagingDirectory = stagingDirectory.mkdirs();
+ if (createdStagingDirectory) {
+ LOG.info("Successfully created staging directory: {}",
stagingDirectory.getAbsolutePath());
+ } else {
+ if (stagingDirectory.exists()) {
+ LOG.info("Staging directory: {} is created by another process",
Review comment:
nit: match the log message at L339, "Successfully created staging
directory by another process: {}"
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -328,6 +330,26 @@ private static InputStream uncheckedInputStream(File file)
{
}
}
+ private void createStagingDirectoryIfNotExists() throws IOException,
SecurityException {
+ if (!stagingDirectory.exists()) {
+ LOG.info("Staging directory: {} does not exist, trying to create one",
+ stagingDirectory.getAbsolutePath());
+ boolean createdStagingDirectory = stagingDirectory.mkdirs();
+ if (createdStagingDirectory) {
+ LOG.info("Successfully created staging directory: {}",
stagingDirectory.getAbsolutePath());
+ } else {
+ if (stagingDirectory.exists()) {
+ LOG.info("Staging directory: {} is created by another process",
+ stagingDirectory.getAbsolutePath());
+ } else {
+ throw new IOException(String
+ .format("Staging directory: %s creation failed due to some
unknown reason",
Review comment:
nit: match the log message at L346, "Fail to create staging directory
due to some unknown reason: {}"
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -170,11 +171,12 @@ public void write(byte[] b, int off, int len) throws
IOException {
}
}
- private void newStream() throws IOException {
+ private void newStream() throws IOException, SecurityException {
Review comment:
Thanks, can you also remove the `SecurityException` after `throws`? We
don't need to throw runtime exception explicitly, just need to document it at
top level which you already did.
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -86,7 +86,8 @@
private boolean closed = false;
@SuppressWarnings("StaticAssignmentInConstructor")
- S3OutputStream(S3Client s3, S3URI location, AwsProperties awsProperties)
throws IOException {
+ S3OutputStream(S3Client s3, S3URI location, AwsProperties awsProperties)
+ throws IOException, SecurityException {
Review comment:
same comment as below, remove SecurityException after throws
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
