damccorm commented on code in PR #29564:
URL: https://github.com/apache/beam/pull/29564#discussion_r1420981406
##########
sdks/python/apache_beam/ml/transforms/embeddings/vertex_ai_test.py:
##########
@@ -44,7 +46,9 @@
VertexAITextEmbeddings is None, 'Vertex AI Python SDK is not installed.')
class VertexAIEmbeddingsTest(unittest.TestCase):
def setUp(self) -> None:
- self.artifact_location = tempfile.mkdtemp()
+ self.artifact_location = tempfile.mkdtemp(prefix='_vertex_ai_test')
+ self.gcs_artifact_location = os.path.join(
+ 'gs://apache-beam-ml/testing/vertex_ai', uuid.uuid4().hex)
Review Comment:
Should we be removing these in tearDown?
##########
sdks/python/apache_beam/ml/transforms/base.py:
##########
@@ -398,6 +398,17 @@ def save_attributes(
artifact_location,
**kwargs,
):
+ # if an artifact location is present, instead of overwriting the
+ # existing file, raise an error since the same artifact location
+ # can be used by multiple beam jobs and this could result in undesired
+ # behavior.
+ if FileSystems.exists(FileSystems.join(artifact_location,
+ _ATTRIBUTE_FILE_NAME)):
+ raise FileExistsError(
+ "The artifact location %s already exists and contains %s. Please "
+ "specify a different location." %
+ (artifact_location, _ATTRIBUTE_FILE_NAME))
Review Comment:
Good call - one possible future enhancement would be to support an overwrite
argument that allows users to do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
